• Hadoop分布式文件系统

    目录

    一、Hadoop

    1、Hadoop简介

    1.1、Hadoop发展简史

    1.2、Hadoop特性优点

    1.3、Hadoop应用

    1.4、Hadoop发行版本

    1.5、Hadoop架构变迁（1.0-2.0变迁）

    2、Hadoop安装部署

    2.1、Hadoop集群概述

    2.2、Hadoop集群模式安装

    2.2.1、集群角色规划：

    2.2.2、服务器基础环境准备

    2.2.3、Hadoop安装说明

    2.2.4、集群启停&Web UI页面

    手动逐个进程启停

    shell脚本一键启停

    进程状态、日志查看

    Web UI

    3、HDFS操作

    3.1、shell命令操作

    3.2、Web UI页面操作

    4、MapReduce+YARN操作

    一、Hadoop

    Hadoop之父Doug Cutting

    Hadoop的发音[hædu:p] ，Cutting儿子对玩具小象的昵称

    狭义上Hadoop指的是Apache软件基金会的一款开源软件。用java语言实现，开源。允许用户使用简单的编程模型实现跨机器集群对海量数据进行分布式计算处理。

    Hadoop核心组件

        Hadoop HDFS（分布式文件存储系统）：解决海量数据存储

        Hadoop YARN（集群资源管理和任务调度框架）：解决资源任务调度

        Hadoop MapReduce（分布式计算框架）：解决海量数据计算

    官网：Apache Hadoop

    广义上Hadoop指的是围绕Hadoop打造的大数据生态圈。

    1、Hadoop简介

        Hadoop起源于Apache Lucene子项目：Nutch

        Nutch的设计目的是构建一个大型的全网搜索引擎。遇到瓶颈：如何解决数十亿网页的存储和索引问题。

    Google三篇论文

        1、《The Google file system》：谷歌分布式文件系统GFS

        2、《MapReduce:Simplified Data Processing on Large Clusters》：谷歌分布式计算框架MapReduce

        3、《Bigtable：A Distributed Storage System for Structured Data》：谷歌结构化数据存储系统

    Hadoop现状

        1、HDFS作为分布式文件存储系统，处在生态圈的底层与核心地位

        2、YARN作为分布式通用的集群资源管理系统和任务调度平台，支撑各种计算引擎运行，保证了Hadoop地位。

        3、MapReduce作为大数据生态圈第一代分布式计算引擎，由于自身设计的模型所产生的弊端，导致企业一线几乎不再直接使用MapReduce进行编程处理，但是很多软件的底层依然在使用MapReduce引擎来处理数据。

    1.1、Hadoop发展简史

    1. 2002年10月，Doug Cutting和Mike Cafarella创建了开源网页爬虫项目Nutch。
    2. 2003年10月，Google发表Google File System论文。
    3. 2004年7月，Doug Cutting和Mike Cafarella在Nutch中实现了类似GFS的功能，即后来HDFS的前身。
    4. 2004年10月，Google发表了MapReduce论文。
    5. 2005年2月，Mike Cafarella在Nutch中实现了MapReduce的最初版本。
    6. 2005年12月，开源搜索项目Nutch移植到新框架，使用MapReduce和HDFS在20个节点稳定运行。
    7. 2006年1月，Doug Cutting加入雅虎，Yahoo!提供一个专门的团队和资源将Hadoop发展成一个可在网络上运行的系统。
    8. 2006年2月，Apache Hadoop项目正式启动以支持MapReduce和HDFS的独立发展。
    9. 2006年3月，Yahoo!建设了第一个Hadoop集群用于开发。
    10. 2006年4月，第一个Apache Hadoop发布。
    11. 2006年11月，Google发表了Bigtable论文，激起了Hbase的创建。
    12. 2007年10月，第一个Hadoop用户组会议召开，社区贡献开始急剧上升。
    13. 2007年，百度开始使用Hadoop做离线处理。
    14. 2007年，中国移动开始在“大云”研究中使用Hadoop技术。
    15. 2008年，淘宝开始投入研究基于Hadoop的系统——云梯，并将其用于处理电子商务相关数据。
    16. 2008年1月，Hadoop成为Apache顶级项目。
    17. 2008年2月，Yahoo!运行了世界上最大的Hadoop应用，宣布其搜索引擎产品部署在一个拥有1万个内核的Hadoop集群上。
    18. 2008年4月，在900个节点上运行1TB排序测试集仅需209秒，成为世界最快。
    19. 2008年8月，第一个Hadoop商业化公司Cloudera成立。
    20. 2008年10月，研究集群每天装载10TB的数据。
    21. 2009 年3月，Cloudera推出世界上首个Hadoop发行版——CDH（Cloudera's Distribution including Apache Hadoop）平台，完全由开放源码软件组成。   Cloudera公司网址
    22. 2009年6月，Cloudera的工程师Tom White编写的《Hadoop权威指南》初版出版，后被誉为Hadoop圣经。
    23. 2009年7月 ，Hadoop Core项目更名为Hadoop Common;
    24. 2009年7月 ，MapReduce 和 Hadoop Distributed File System (HDFS) 成为Hadoop项目的独立子项目。
    25. 2009年8月，Hadoop创始人Doug Cutting加入Cloudera担任首席架构师。
    26. 2009年10月，首届Hadoop World大会在纽约召开。
    27. 2010年5月，IBM提供了基于Hadoop 的大数据分析软件——InfoSphere BigInsights，包括基础版和企业版。
    28. 2011年3月，Apache Hadoop获得Media Guardian Innovation Awards媒体卫报创新奖
    29. 2012年3月，企业必须的重要功能HDFS NameNode HA被加入Hadoop主版本。
    30. 2012年8月，另外一个重要的企业适用功能YARN成为Hadoop子项目。
    31. 2014年2月，Spark逐渐代替MapReduce成为Hadoop的缺省执行引擎，并成为Apache基金会顶级项目。
    32. 2017年12月，Release 3.0.0 generally available

    hadoop官网         Cloudera公司网址

    1.2、Hadoop特性优点

        1、扩容能力：Hadoop是在可用的计算机集群间分配数据并完成计算任务的，这些集群可方便灵活的方式扩展到数以千计的节点。

        2、成本低：Hadoop集群允许通过部署普通廉价的机器组成集群来处理大数据，以至于成本很低。看重的是集群整体能力。

        3、效率高：通过并发数据，Hadoop可以在节点之间动态并行的移动数据，使得速度非常快。

        4、可靠性：能自动维护数据的多份复制，并且在任务失败后能自动地重新部署（redeploy）计算任务。所以Hadoop的按位存储和处理数据的能力值得人们信赖。

    Hadoop成功的魅力--通用性

    Hadoop成功的魅力--简单

    1.3、Hadoop应用

    Yahoo：支持广告系统、用户行为分析、反垃圾邮件系统

    Facebook：存储处理数据挖掘和日志统计、构建基于Hadoop数据仓库平台（Hive来自FB）

    IBM：蓝云基础设施构建

    百度：数据分析和挖掘 竞价排名

    阿里巴巴：交易数据 信用分析

    腾讯：用户关系数据

    华为：对Hadoop的HA方案，以及HBase领域有深入的研究

    1.4、Hadoop发行版本

    开源社区版：Apache开源社区发行，也是官方发行版本。

        优点：更新迭代快 缺点：兼容稳定性不周

    商业发行版：商业公司发行 基于Apache开源协议某些服务需要收费

        优点：稳定兼容好 缺点：收费 版本更新慢

    Apache开源社区版本

        Apache Hadoop

    商业发行版本

        Cloudera：Apache Hadoop open source ecosystem | Cloudera

        Hortonworks：Hortonworks Data Platform | Cloudera

    1.5、Hadoop架构变迁（1.0-2.0变迁）

    Hadoop1.0

        HDFS（分布式文件存储）

        MapReduce（资源管理和分布式数据处理）

    Hadoop2.0

        HDFS（分布式文件存储）

        MapReduce（分布式数据处理）

        YARN（集群资源管理、任务调度）

    Hadoop3.0

        Hadoop 3.0架构组件和Hadoop 2.0类似，3.0着重于性能优化。

        通用方面：精简内核、类路径隔离、shell脚本重构

        Hadoop HDFS：EC纠删码、多NameNode支持

        Hadoop MapReduce：任务本地化优化、内存参数自动推断

        Hadoop YARN：Timeline Service V2、队列配置

    2、Hadoop安装部署

    2.1、Hadoop集群概述

    Hadoop集群包括两个集群：HDFS集群、YARN集群

    两个集群逻辑上分离、通常物理上在一起

    两个集群都是标准的主从架构集群

    HDFS集群（分布式存储）

        主角色：NameNode

        从角色：DataNode

        主角色辅助角色：SecondaryNameNode

    YARN集群（资源管理、调度）

        主角色：RosoureManager

        从角色：NodeManager

    逻辑上分离：两个集群互相之间没有依赖，互不影响

    物理上在一起：某些角色进程往往部署在同一台物理服务器上

    MapReduce集群呢？MapReduce是计算框架，代码层面的组件，没有集群之说

    2.2、Hadoop集群模式安装

    官网下载

        hadoop-3.3.6-src.tar.gz  :源码包

        hadoop-3.3.6.tar.gz：官方编译安装包

    为什么要重新编译Hadoop源码？

        匹配不同操作系统本地库环境，Hadoop某些操作比如压缩、IO需要调用系统本地库（*.so|*.dll） 修改源码、重构源码

    如何编译Hadoop？

        源码包根目录下文件：BUILDING.txt

    2.2.1、集群角色规划：

        资源上有抢夺冲突的，尽量不要部署在一起

        工作上需要相互配合的，尽量部署在一起

        依赖内存工作的NameNode是不是部署在大内存机器上

    服务器

    		运行角色
    node1namenode datanode resourcemanager nodemanager
    node2secondarynamenode datanode  nodemanager
    node3datanode  nodemanager
    2.2.2、服务器基础环境准备

    1、主机名（3台机器）

    vi /etc/hostname
    1. [root@node1 ~]# cat /etc/hostname
    2. node1.lwz.cn

    2、Host映射（3台机器）

    vi /etc/hosts
    1. [root@node1 ~]# cat /etc/hosts
    2. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    3. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    4.  
    5. 192.168.18.81  node1 node1.lwz.cn
    6. 192.168.18.82  node2 node2.lwz.cn
    7. 192.168.18.83  node3 node3.lwz.cn

    3、关闭防火墙

    1. systemctl stop firewalld.service  #关闭防火墙
    2. systemctl disable firewalld.service  #禁止防火墙开机自启

    4、配置SSH免密登录（node1执行->node|node2|node3）

    1. ssh-keygen  #4个回车 生成公钥、私钥
    2. shh-copy-id node1、shh-copy-id node2、shh-copy-id node3 #

    Linux CentOS 7

    5、集群时间同步（3台机器）

    1. yum -y install ntpdate
    2.  
    3. ntpdate ntp4.aliyun.com    #集群时间同步
    4.  
    5. [root@node1 ~]# ntpdate ntp4.aliyun.com
    6. 25 Dec 23:36:11 ntpdate[9578]: adjust time server 203.107.6.88 offset 0.164550 sec
    7. [root@node1 ~]# date
    8. Mon Dec 25 23:37:34 CST 2023

    6、创建统一工作目录（3台机器）

    1. mkdir -p /export/server/       #软件安装路径
    2. mkdir -p /export/data/         #数据存储路径
    3. mkdir -p /export/software/     #安装包存放路径

    7、JDK1.8安装

    Linux 软件安装

    8、上传，解压Hadoop安装包

    [root@node1 hadoop-3.3.6]# tar zxvf hadoop-3.3.6.tar.gz
    2.2.3、Hadoop安装说明

    Hadoop安装包目录

    目录说明
    binHadoop最基本的管理脚本和使用脚本的目录，这些脚本是sbin目录下管理脚本的基础实现，用户可以直接使用这些脚本管理和使用Hadoop
    etcHadoop配置文件所在的目录
    include对外提供的编程库头文件（具体动态库和静态库在lib目录中），这些头文件均是用C++定义的，通常用于C++程序访问HDFS或者编写MapReduce程序。
    lib该目录包含了Hadoop对外提供的编程动态库和静态库，与include目录中的头文件结合使用。
    libexec各个服务对用的shell配置文件所在的目录，可用于配置日志输出、启动参数（比如JVM参数）等基本信息。
    sbinHadoop管理脚本所在的目录，主要包含HDFS和YARN中各类服务的启动/关闭脚本。
    shareHadoop各个模块编译后的jar包所在的目录，官方自带示例。

    配置文件概述

    官网文档：Hadoop – Apache Hadoop 3.3.6

    第一类1个：hadoop-env.sh

    第二类4个：xxxx-site.xml ，site表示的是用户定义的配置，会覆盖default中的默认配置。

                        core-site.xml    核心模块配置

                        hdfs-site.xml    hdfs文件系统模块配置

                        mapred-site.xml    MapReduce模块配置

                        yarn-site.xml    yarn模块配置

    第三类1个：workers

    所有的配置文件目录：/export/server/hadoop-3.3.6/etc/hadoop

    修改配置文件

    hadoop-env.sh

    1. 文件最后添加
    2. export JAVA_HOME=/export/server/jdk1.8.0_202
    3.  
    4. export HDFS_NAMENODE_USER=root
    5. export HDFS_DATANODE_USER=root
    6. export HDFS_SECONDARYNAMENODE_USER=root
    7. export YARN_RESOURCEMANAGER_USER=root
    8. export YARN_NODEMANAGER_USER=root

    core-site.xml    拷贝到中间

    2. <property>
    3.    <name>fs.defaultFSname>
    4.    <value>hdfs://node1:8020value>
    5. property>
    6.  
    8. <property>
    9.    <name>hadoop.tmp.dirname>
    10.    <value>/export/data/hadoop-3.3.6value>
    11. property>
    12.  
    14. <property>
    15.    <name>hadoop.http.staticuser.username>
    16.    <value>rootvalue>
    17. property>
    18.  
    20. <property>
    21.    <name>hadoop.proxyuser.root.hostsname>
    22.    <value>*value>
    23. property>
    24.  
    25. <property>
    26.    <name>hadoop.proxyuser.root.groupsname>
    27.    <value>*value>
    28. property>
    29.  
    31. <property>
    32.    <name>fs.trash.intervalname>
    33.    <value>1440value>
    34. property>

    hdfs-site.xml 拷贝到中间

    2. <property>
    3.    <name>dfs.namenode.secondary.http-addressname>
    4.    <value>node2:9868value>
    5. property>

    mapred-site.xml 拷贝到中间

    2. <property>
    3.    <name>mapreduce.framework.namename>
    4.    <value>yarnvalue>
    5. property>
    6.  
    8. <property>
    9.    <name>mapreduce.jobhistory.addressname>
    10.    <value>node1:10020value>
    11. property>
    12.  
    14. <property>
    15.    <name>mapreduce.jobhistory.webapp.addressname>
    16.    <value>node1:19888value>
    17. property>
    18.  
    19. <property>
    20.    <name>yarn.app.mapreduce.am.envname>
    21.    <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}value>
    22. property>
    23.  
    24. <property>
    25.    <name>mapreduce.map.envname>
    26.    <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}value>
    27. property>
    28.  
    29. <property>
    30.    <name>mapreduce.reduce.envname>
    31.    <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}value>
    32. property>

    yarn-site.xml  拷贝到中间

    2. <property>
    3.    <name>yarn.resourcemanager.hostnamename>
    4.    <value>node1value>
    5. property>
    6.  
    7. <property>
    8.    <name>yarn.nodemanager.aux-servicesname>
    9.    <value>mapreduce_shufflevalue>
    10. property>
    11.  
    13. <property>
    14.    <name>yarn.nodemanager.pmem-check-enabledname>
    15.    <value>falsevalue>
    16. property>
    17.  
    19. <property>
    20.    <name>yarn.nodemanager.vmem-check-enabledname>
    21.    <value>falsevalue>
    22. property>
    23.  
    25. <property>
    26.    <name>yarn.log-aggregation-enablename>
    27.    <value>truevalue>
    28. property>
    29.  
    31. <property>
    32.    <name>yarn.log.server.urlname>
    33.    <value>http://node1:19888/jobhistory/logsvalue>
    34. property>
    35.  
    37. <property>
    38.    <name>yarn.log-aggregation.retain-secondsname>
    39.    <value>604800value>
    40. property>

    workers  把原来localhost内容删除，拷贝下面内容保存

    1. node1.lwz.cn
    2. node2.lwz.cn
    3. node3.lwz.cn

    分发同步hadoop安装包

    1. cd /export/server
    2.  
    3. scp -r hadoop-3.3.6 root@node2:$PWD
    4. scp -r hadoop-3.3.6 root@node3:$PWD

    将hadoop添加到环境变量（3台机器都做）

    1. vi /etc/profile
    2.  
    3. export HADOOP_HOME=/export/server/hadoop-3.3.6
    4. export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
    5.  
    6. source /etc/profile
    7.

    Hadoop集群启动

    (首次启动)初始化namenode

        首次启动HDFS时，必须对其进行格式化操作。

        format本质上是初始化工作，进行HDFS清理和准备工作，只能执行一次。集群启动前执行，在主机器上执行。

        如果多次format除了造成数据丢失外，还会导致hdfs集群主从角色之间互不识别。通过删除所有机器hadoop.tmp.dir目录重新format解决

    hdfs namenode -format
    1. [root@node1 hadoop]# hdfs namenode -format
    2. 2023-12-27 00:24:35,400 INFO namenode.NameNode: STARTUP_MSG:
    3. /************************************************************
    4. STARTUP_MSG: Starting NameNode
    5. STARTUP_MSG:   host = node1/192.168.18.81
    6. STARTUP_MSG:   args = [-format]
    7. STARTUP_MSG:   version = 3.3.6
    8. STARTUP_MSG:   classpath = /export/server/hadoop-3.3.6/etc/hadoop:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-resolver-dns-native-macos-4.1.89.Final-osx-aarch_64.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-socks-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-math3-3.1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jetty-xml-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-xml-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-sctp-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-collections-3.2.2.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jsp-api-2.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-codec-1.15.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/reload4j-1.2.22.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jersey-server-1.19.4.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jsch-0.1.55.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerby-util-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-io-2.8.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/token-provider-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/stax2-api-4.2.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-dns-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/avro-1.7.7.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-text-1.10.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-native-kqueue-4.1.89.Final-osx-aarch_64.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/hadoop-auth-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerby-config-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-lang3-3.12.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jetty-server-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jersey-servlet-1.19.4.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerb-common-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/guava-27.0-jre.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-rxtx-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jersey-core-1.19.4.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-logging-1.1.3.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/curator-recipes-5.2.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-resolver-dns-classes-macos-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-handler-ssl-ocsp-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-native-epoll-4.1.89.Final-linux-aarch_64.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/curator-framework-5.2.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jul-to-slf4j-1.7.36.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-http-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-http2-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-haproxy-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/checker-qual-2.5.2.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jsr305-3.0.2.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerb-server-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jetty-util-ajax-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerb-client-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/slf4j-api-1.7.36.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-handler-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/zookeeper-3.6.3.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jetty-util-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-configuration2-2.8.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jersey-json-1.20.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-udt-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-compress-1.21.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-memcache-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-cli-1.2.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jettison-1.5.4.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-smtp-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-native-epoll-4.1.89.Final-linux-x86_64.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jetty-http-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-common-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-daemon-1.0.13.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/httpcore-4.4.13.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/failureaccess-1.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jakarta.activation-api-1.2.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-stomp-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-all-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/metrics-core-3.2.4.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-native-unix-common-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-resolver-dns-native-macos-4.1.89.Final-osx-x86_64.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jackson-core-2.12.7.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jetty-servlet-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/dnsjava-2.1.7.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/zookeeper-jute-3.6.3.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-redis-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jetty-webapp-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerb-util-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jackson-annotations-2.12.7.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/hadoop-annotations-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jackson-databind-2.12.7.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-handler-proxy-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/paranamer-2.3.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerb-core-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/gson-2.9.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jetty-security-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-native-kqueue-4.1.89.Final-osx-x86_64.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/re2j-1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/httpclient-4.5.13.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/curator-client-5.2.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-classes-epoll-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-resolver-dns-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-resolver-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jetty-io-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/commons-net-3.9.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/snappy-java-1.1.8.2.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/woodstox-core-5.4.0.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-classes-kqueue-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-mqtt-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/nimbus-jose-jwt-9.8.1.jar:/export/server/hadoop-3.3.6/share/hadoop/common/lib/netty-buffer-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/common/hadoop-common-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/common/hadoop-nfs-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/common/hadoop-common-3.3.6-tests.jar:/export/server/hadoop-3.3.6/share/hadoop/common/hadoop-registry-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/common/hadoop-kms-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-resolver-dns-native-macos-4.1.89.Final-osx-aarch_64.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-socks-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/HikariCP-java7-2.4.12.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-xml-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-xml-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-sctp-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-codec-1.15.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/reload4j-1.2.22.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jersey-server-1.19.4.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-io-2.8.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kotlin-stdlib-1.4.10.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/stax2-api-4.2.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-dns-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/avro-1.7.7.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-text-1.10.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-native-kqueue-4.1.89.Final-osx-aarch_64.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/hadoop-auth-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-lang3-3.12.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-server-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jersey-servlet-1.19.4.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-rxtx-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jersey-core-1.19.4.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/curator-recipes-5.2.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-resolver-dns-classes-macos-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-handler-ssl-ocsp-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/okio-2.8.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-native-epoll-4.1.89.Final-linux-aarch_64.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/curator-framework-5.2.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-http-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-http2-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-haproxy-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-handler-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/zookeeper-3.6.3.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/hadoop-shaded-guava-1.1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-util-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-configuration2-2.8.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jersey-json-1.20.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-udt-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-compress-1.21.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-memcache-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jettison-1.5.4.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-smtp-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-native-epoll-4.1.89.Final-linux-x86_64.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-http-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-common-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/httpcore-4.4.13.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jakarta.activation-api-1.2.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-stomp-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-all-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/metrics-core-3.2.4.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-native-unix-common-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-resolver-dns-native-macos-4.1.89.Final-osx-x86_64.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jackson-core-2.12.7.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-servlet-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/re2j-1.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/zookeeper-jute-3.6.3.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-redis-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-webapp-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jackson-annotations-2.12.7.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/hadoop-annotations-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jackson-databind-2.12.7.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-handler-proxy-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/paranamer-2.3.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/gson-2.9.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-security-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-native-kqueue-4.1.89.Final-osx-x86_64.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/httpclient-4.5.13.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/curator-client-5.2.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-classes-epoll-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-resolver-dns-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-resolver-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-io-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/okhttp-4.9.3.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-net-3.9.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/snappy-java-1.1.8.2.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/woodstox-core-5.4.0.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-classes-kqueue-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-mqtt-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/nimbus-jose-jwt-9.8.1.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-buffer-4.1.89.Final.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/lib/kotlin-stdlib-common-1.4.10.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.6-tests.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.6-tests.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-3.3.6-tests.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-client-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-nfs-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-client-3.3.6-tests.jar:/export/server/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.6-tests.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/java-util-1.9.0.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/objenesis-2.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jakarta.xml.bind-api-2.3.2.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/fst-2.50.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/websocket-client-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jersey-client-1.19.4.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/bcpkix-jdk15on-1.68.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.12.7.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/asm-tree-9.4.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/javax.inject-1.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/snakeyaml-2.0.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/javax-websocket-server-impl-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/websocket-api-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/asm-commons-9.4.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/bcprov-jdk15on-1.68.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jackson-jaxrs-base-2.12.7.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/javax-websocket-client-impl-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jline-3.9.0.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/websocket-server-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/javax.websocket-api-1.0.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/javax.websocket-client-api-1.0.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jetty-plus-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jetty-client-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jetty-annotations-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/websocket-servlet-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/json-io-2.5.1.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jetty-jndi-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/aopalliance-1.0.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/websocket-common-9.4.51.v20230217.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jersey-guice-1.19.4.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jna-5.2.0.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.12.7.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/lib/guice-4.0.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-registry-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-applications-mawo-core-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-client-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-router-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-tests-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-services-core-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-services-api-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-common-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-common-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.3.6.jar:/export/server/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-api-3.3.6.jar
    9. STARTUP_MSG:   build = https://github.com/apache/hadoop.git -r 1be78238728da9266a4f88195058f08fd012bf9c; compiled by 'ubuntu' on 2023-06-18T08:22Z
    10. STARTUP_MSG:   java = 1.8.0_202
    11. ************************************************************/
    12. 2023-12-27 00:24:35,410 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
    13. 2023-12-27 00:24:35,521 INFO namenode.NameNode: createNameNode [-format]
    14. 2023-12-27 00:24:36,139 INFO namenode.NameNode: Formatting using clusterid: CID-4dc541c7-8aa2-499a-bea5-69f139b26977
    15. 2023-12-27 00:24:36,182 INFO namenode.FSEditLog: Edit logging is async:true
    16. 2023-12-27 00:24:36,210 INFO namenode.FSNamesystem: KeyProvider: null
    17. 2023-12-27 00:24:36,211 INFO namenode.FSNamesystem: fsLock is fair: true
    18. 2023-12-27 00:24:36,212 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
    19. 2023-12-27 00:24:36,234 INFO namenode.FSNamesystem: fsOwner                = root (auth:SIMPLE)
    20. 2023-12-27 00:24:36,234 INFO namenode.FSNamesystem: supergroup             = supergroup
    21. 2023-12-27 00:24:36,235 INFO namenode.FSNamesystem: isPermissionEnabled    = true
    22. 2023-12-27 00:24:36,235 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true
    23. 2023-12-27 00:24:36,235 INFO namenode.FSNamesystem: HA Enabled: false
    24. 2023-12-27 00:24:36,283 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
    25. 2023-12-27 00:24:36,445 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit : configured=1000, counted=60, effected=1000
    26. 2023-12-27 00:24:36,445 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
    27. 2023-12-27 00:24:36,449 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
    28. 2023-12-27 00:24:36,450 INFO blockmanagement.BlockManager: The block deletion will start around 2023 Dec 27 00:24:36
    29. 2023-12-27 00:24:36,451 INFO util.GSet: Computing capacity for map BlocksMap
    30. 2023-12-27 00:24:36,451 INFO util.GSet: VM type       = 64-bit
    31. 2023-12-27 00:24:36,452 INFO util.GSet: 2.0% max memory 1.7 GB = 34.8 MB
    32. 2023-12-27 00:24:36,453 INFO util.GSet: capacity      = 2^22 = 4194304 entries
    33. 2023-12-27 00:24:36,463 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
    34. 2023-12-27 00:24:36,463 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
    35. 2023-12-27 00:24:36,470 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999
    36. 2023-12-27 00:24:36,470 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
    37. 2023-12-27 00:24:36,470 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
    38. 2023-12-27 00:24:36,471 INFO blockmanagement.BlockManager: defaultReplication         = 3
    39. 2023-12-27 00:24:36,471 INFO blockmanagement.BlockManager: maxReplication             = 512
    40. 2023-12-27 00:24:36,471 INFO blockmanagement.BlockManager: minReplication             = 1
    41. 2023-12-27 00:24:36,471 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
    42. 2023-12-27 00:24:36,471 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
    43. 2023-12-27 00:24:36,471 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
    44. 2023-12-27 00:24:36,471 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
    45. 2023-12-27 00:24:36,537 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
    46. 2023-12-27 00:24:36,538 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
    47. 2023-12-27 00:24:36,538 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
    48. 2023-12-27 00:24:36,538 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
    49. 2023-12-27 00:24:36,578 INFO util.GSet: Computing capacity for map INodeMap
    50. 2023-12-27 00:24:36,578 INFO util.GSet: VM type       = 64-bit
    51. 2023-12-27 00:24:36,578 INFO util.GSet: 1.0% max memory 1.7 GB = 17.4 MB
    52. 2023-12-27 00:24:36,578 INFO util.GSet: capacity      = 2^21 = 2097152 entries
    53. 2023-12-27 00:24:36,579 INFO namenode.FSDirectory: ACLs enabled? true
    54. 2023-12-27 00:24:36,579 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
    55. 2023-12-27 00:24:36,579 INFO namenode.FSDirectory: XAttrs enabled? true
    56. 2023-12-27 00:24:36,580 INFO namenode.NameNode: Caching file names occurring more than 10 times
    57. 2023-12-27 00:24:36,585 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
    58. 2023-12-27 00:24:36,587 INFO snapshot.SnapshotManager: SkipList is disabled
    59. 2023-12-27 00:24:36,591 INFO util.GSet: Computing capacity for map cachedBlocks
    60. 2023-12-27 00:24:36,591 INFO util.GSet: VM type       = 64-bit
    61. 2023-12-27 00:24:36,592 INFO util.GSet: 0.25% max memory 1.7 GB = 4.3 MB
    62. 2023-12-27 00:24:36,592 INFO util.GSet: capacity      = 2^19 = 524288 entries
    63. 2023-12-27 00:24:36,600 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
    64. 2023-12-27 00:24:36,601 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
    65. 2023-12-27 00:24:36,601 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
    66. 2023-12-27 00:24:36,605 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
    67. 2023-12-27 00:24:36,605 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
    68. 2023-12-27 00:24:36,607 INFO util.GSet: Computing capacity for map NameNodeRetryCache
    69. 2023-12-27 00:24:36,607 INFO util.GSet: VM type       = 64-bit
    70. 2023-12-27 00:24:36,607 INFO util.GSet: 0.029999999329447746% max memory 1.7 GB = 534.2 KB
    71. 2023-12-27 00:24:36,607 INFO util.GSet: capacity      = 2^16 = 65536 entries
    72. 2023-12-27 00:24:36,632 INFO namenode.FSImage: Allocated new BlockPoolId: BP-626243755-192.168.18.81-1703607876623
    73. 2023-12-27 00:24:36,653 INFO common.Storage: Storage directory /export/data/hadoop-3.3.6/dfs/name has been successfully formatted.
    74. 2023-12-27 00:24:36,680 INFO namenode.FSImageFormatProtobuf: Saving image file /export/data/hadoop-3.3.6/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
    75. 2023-12-27 00:24:36,837 INFO namenode.FSImageFormatProtobuf: Image file /export/data/hadoop-3.3.6/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 399 bytes saved in 0 seconds .
    76. 2023-12-27 00:24:36,855 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
    77. 2023-12-27 00:24:36,882 INFO namenode.FSNamesystem: Stopping services started for active state
    78. 2023-12-27 00:24:36,882 INFO namenode.FSNamesystem: Stopping services started for standby state
    79. 2023-12-27 00:24:36,887 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
    80. 2023-12-27 00:24:36,887 INFO namenode.NameNode: SHUTDOWN_MSG:
    81. /************************************************************
    82. SHUTDOWN_MSG: Shutting down NameNode at node1/192.168.18.81
    83. ************************************************************/
    84. [root@node1 hadoop]#

    2023-12-27 00:24:36,653 INFO common.Storage: Storage directory /export/data/hadoop-3.3.6/dfs/name has been successfully formatted.   代表成功

    2.2.4、集群启停&Web UI页面
    手动逐个进程启停

        每台机器上每次手动启动关闭一个角色进程，可以精准控制每个进程启停，避免群起群停。

    HDFS集群

    1. #hadoop2.x版本命令
    2. hadoop-daemon.sh start|stop namenode|datanode|secondarynamenode
    3. #hadoop3.x版本命令
    4. hdfs --daemon start|stop namenode|datanode|secondarynamenode

    YARN集群

    1. #hadoop2.x版本命令
    2. yarn-daemon.sh start|stop resourcemanager|nodemanager
    3. #hadoop3.x版本命令
    4. yarn --daemon start|stop resourcemanager|nodemanager
    shell脚本一键启停

        在node1上，使用软件自带的shell脚本一键启动。前提：配置好机器之间的SSH免密登录和workers文件。

    HDFS集群

        start-dfs.sh、stop-dfs.sh

    1. ####################################node1
    2. [root@node1 ~]# start-dfs.sh
    3. Starting namenodes on [node1]
    4. Last login: Tue Jan  2 22:33:42 CST 2024 from 192.168.18.69 on pts/0
    5. Starting datanodes
    6. Last login: Tue Jan  2 22:48:37 CST 2024 on pts/0
    7. node3.lwz.cn: Warning: Permanently added 'node3.lwz.cn' (ECDSA) to the list of known hosts.
    8. node1.lwz.cn: Warning: Permanently added 'node1.lwz.cn' (ECDSA) to the list of known hosts.
    9. node2.lwz.cn: Warning: Permanently added 'node2.lwz.cn' (ECDSA) to the list of known hosts.
    10. node2.lwz.cn: WARNING: /export/server/hadoop-3.3.6/logs does not exist. Creating.
    11. node3.lwz.cn: WARNING: /export/server/hadoop-3.3.6/logs does not exist. Creating.
    12. Starting secondary namenodes [node2]
    13. Last login: Tue Jan  2 22:48:40 CST 2024 on pts/0
    14. [root@node1 ~]# jps
    15. 1604 NameNode
    16. 1732 DataNode
    17. 2045 Jps
    18. [root@node1 ~]#
    19. ####################################node2
    20. [root@node2 ~]# jps
    21. 1543 SecondaryNameNode
    22. 1480 DataNode
    23. 1630 Jps
    24. [root@node2 ~]#
    25. ####################################node3
    26. [root@node3 ~]# jps
    27. 1476 DataNode
    28. 1547 Jps
    29. [root@node3 ~]#

    YARN集群

        start-yarn.sh、stop-yarn.sh

    1. ####################################node1
    2. [root@node1 ~]# start-yarn.sh
    3. Starting resourcemanager
    4. Last login: Tue Jan  2 22:48:44 CST 2024 on pts/0
    5. Starting nodemanagers
    6. Last login: Tue Jan  2 22:51:50 CST 2024 on pts/0
    7. [root@node1 ~]# jps
    8. 2673 Jps
    9. 2322 NodeManager
    10. 1604 NameNode
    11. 1732 DataNode
    12. 2187 ResourceManager
    13. [root@node1 ~]#
    14. ####################################node2
    15. [root@node2 ~]# jps
    16. 1696 NodeManager
    17. 1543 SecondaryNameNode
    18. 1480 DataNode
    19. 1802 Jps
    20. ####################################node3
    21. [root@node3 ~]# jps
    22. 1476 DataNode
    23. 1607 NodeManager
    24. 1710 Jps

    Hadoop集群

        start-all.sh、stop-all.sh

    进程状态、日志查看

    1、启动完毕之后可以使用jps命令查看进程是否启动成功

    2、Hadoop启动日志路径：/export/server/hadoop-3.3.6/logs

    1. [root@node1 ~]# cd /export/server/hadoop-3.3.6/logs
    2. [root@node1 logs]# ll
    3. total 212
    4. -rw-r--r--. 1 root root 41996 Jan  2 22:48 hadoop-root-datanode-node1.lwz.cn.log
    5. -rw-r--r--. 1 root root   692 Jan  2 22:48 hadoop-root-datanode-node1.lwz.cn.out
    6. -rw-r--r--. 1 root root 51244 Jan  2 22:49 hadoop-root-namenode-node1.lwz.cn.log
    7. -rw-r--r--. 1 root root   692 Jan  2 22:48 hadoop-root-namenode-node1.lwz.cn.out
    8. -rw-r--r--. 1 root root 46338 Jan  2 22:52 hadoop-root-nodemanager-node1.lwz.cn.log
    9. -rw-r--r--. 1 root root  2210 Jan  2 22:52 hadoop-root-nodemanager-node1.lwz.cn.out
    10. -rw-r--r--. 1 root root 52832 Jan  2 22:52 hadoop-root-resourcemanager-node1.lwz.cn.log
    11. -rw-r--r--. 1 root root  2226 Jan  2 22:51 hadoop-root-resourcemanager-node1.lwz.cn.out
    12. -rw-r--r--. 1 root root     0 Dec 27 00:20 SecurityAuth-root.audit
    13. drwxr-xr-x. 2 root root     6 Jan  2 22:51 userlogs
    Web UI

    HDFS集群

    地址：http://namenode_host:9870

        其中namenode_host是namenode运行所在机器的主机名或者ip

        如果使用主机名访问，别忘了在windows配置hosts

    最常用的是Utilities-->Browse the file system

    YARN集群

    地址：http://resourcemanager_host:8088

        其中resourcemanager_host是resourcemanager运行所在机器的主机名或者ip

        如果使用主机名访问，别忘了在windows配置hosts

    3、HDFS操作

    3.1、shell命令操作

    1. [root@node1 ~]# hadoop fs -mkdir /lwztest
    2. [root@node1 ~]# echo 123456adf > 1.txt
    3. [root@node1 ~]# cat 1.txt
    4. 123456adf
    5. [root@node1 ~]# hadoop fs -put 1.txt /lwztest
    6. [root@node1 ~]# hadoop fs -ls /
    7. Found 1 items
    8. drwxr-xr-x   - root supergroup          0 2024-01-02 23:52 /lwztest
    9. [root@node1 ~]#
    10.

    3.2、Web UI页面操作

    思考：

    1、HDFS本质就是一个文件系统

    2、有目录树结构和Linux类似，分文件，文件夹

    3、为什么上传一个小文件也这么慢？

    4、MapReduce+YARN操作

    案例1：执行Hadoop官方自带的MapReduce案例，评估圆周率π的值

    1. [root@node1 ~]# cd /export/server/hadoop-3.3.6/share/hadoop/mapreduce/
    2. [root@node1 mapreduce]# ls
    3. hadoop-mapreduce-client-app-3.3.6.jar         hadoop-mapreduce-client-jobclient-3.3.6.jar        hadoop-mapreduce-examples-3.3.6.jar
    4. hadoop-mapreduce-client-common-3.3.6.jar      hadoop-mapreduce-client-jobclient-3.3.6-tests.jar  jdiff
    5. hadoop-mapreduce-client-core-3.3.6.jar        hadoop-mapreduce-client-nativetask-3.3.6.jar       lib-examples
    6. hadoop-mapreduce-client-hs-3.3.6.jar          hadoop-mapreduce-client-shuffle-3.3.6.jar          sources
    7. hadoop-mapreduce-client-hs-plugins-3.3.6.jar  hadoop-mapreduce-client-uploader-3.3.6.jar
    8. [root@node1 mapreduce]# hadoop jar hadoop-mapreduce-examples-3.3.6.jar pi 2 4
    9. Number of Maps  = 2
    10. Samples per Map = 4
    11. Wrote input for Map #0
    12. Wrote input for Map #1
    13. Starting Job
    14. 2024-01-03 00:05:35,990 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node1/192.168.18.81:8032
    15. 2024-01-03 00:05:36,427 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_17042071157                                                                                        02_0001
    16. 2024-01-03 00:05:36,633 INFO input.FileInputFormat: Total input files to process : 2
    17. 2024-01-03 00:05:36,739 INFO mapreduce.JobSubmitter: number of splits:2
    18. 2024-01-03 00:05:36,889 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1704207115702_0001
    19. 2024-01-03 00:05:36,889 INFO mapreduce.JobSubmitter: Executing with tokens: []
    20. 2024-01-03 00:05:37,048 INFO conf.Configuration: resource-types.xml not found
    21. 2024-01-03 00:05:37,048 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
    22. 2024-01-03 00:05:37,469 INFO impl.YarnClientImpl: Submitted application application_1704207115702_0001
    23. 2024-01-03 00:05:37,517 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1704207115702_0001/
    24. 2024-01-03 00:05:37,518 INFO mapreduce.Job: Running job: job_1704207115702_0001
    25. 2024-01-03 00:05:44,645 INFO mapreduce.Job: Job job_1704207115702_0001 running in uber mode : false
    26. 2024-01-03 00:05:44,647 INFO mapreduce.Job:  map 0% reduce 0%
    27. 2024-01-03 00:05:49,790 INFO mapreduce.Job:  map 100% reduce 0%
    28. 2024-01-03 00:05:56,882 INFO mapreduce.Job:  map 100% reduce 100%
    29. 2024-01-03 00:05:57,937 INFO mapreduce.Job: Job job_1704207115702_0001 completed successfully
    30. 2024-01-03 00:05:58,074 INFO mapreduce.Job: Counters: 54
    31.         File System Counters
    32.                 FILE: Number of bytes read=50
    33.                 FILE: Number of bytes written=831342
    34.                 FILE: Number of read operations=0
    35.                 FILE: Number of large read operations=0
    36.                 FILE: Number of write operations=0
    37.                 HDFS: Number of bytes read=520
    38.                 HDFS: Number of bytes written=215
    39.                 HDFS: Number of read operations=13
    40.                 HDFS: Number of large read operations=0
    41.                 HDFS: Number of write operations=3
    42.                 HDFS: Number of bytes read erasure-coded=0
    43.         Job Counters
    44.                 Launched map tasks=2
    45.                 Launched reduce tasks=1
    46.                 Data-local map tasks=2
    47.                 Total time spent by all maps in occupied slots (ms)=5594
    48.                 Total time spent by all reduces in occupied slots (ms)=3894
    49.                 Total time spent by all map tasks (ms)=5594
    50.                 Total time spent by all reduce tasks (ms)=3894
    51.                 Total vcore-milliseconds taken by all map tasks=5594
    52.                 Total vcore-milliseconds taken by all reduce tasks=3894
    53.                 Total megabyte-milliseconds taken by all map tasks=5728256
    54.                 Total megabyte-milliseconds taken by all reduce tasks=3987456
    55.         Map-Reduce Framework
    56.                 Map input records=2
    57.                 Map output records=4
    58.                 Map output bytes=36
    59.                 Map output materialized bytes=56
    60.                 Input split bytes=284
    61.                 Combine input records=0
    62.                 Combine output records=0
    63.                 Reduce input groups=2
    64.                 Reduce shuffle bytes=56
    65.                 Reduce input records=4
    66.                 Reduce output records=0
    67.                 Spilled Records=8
    68.                 Shuffled Maps =2
    69.                 Failed Shuffles=0
    70.                 Merged Map outputs=2
    71.                 GC time elapsed (ms)=226
    72.                 CPU time spent (ms)=1710
    73.                 Physical memory (bytes) snapshot=887963648
    74.                 Virtual memory (bytes) snapshot=8386064384
    75.                 Total committed heap usage (bytes)=736624640
    76.                 Peak Map Physical memory (bytes)=314757120
    77.                 Peak Map Virtual memory (bytes)=2793766912
    78.                 Peak Reduce Physical memory (bytes)=260431872
    79.                 Peak Reduce Virtual memory (bytes)=2799820800
    80.         Shuffle Errors
    81.                 BAD_ID=0
    82.                 CONNECTION=0
    83.                 IO_ERROR=0
    84.                 WRONG_LENGTH=0
    85.                 WRONG_MAP=0
    86.                 WRONG_REDUCE=0
    87.         File Input Format Counters
    88.                 Bytes Read=236
    89.         File Output Format Counters
    90.                 Bytes Written=97
    91. Job Finished in 22.166 seconds
    92. Estimated value of Pi is 3.50000000000000000000
    93. [root@node1 mapreduce]#

    案例2：文件单词统计案例

    1. [root@node1 mapreduce]# hadoop fs -mkdir -p /wordcount/input
    2. [root@node1 mapreduce]# echo hi lwz i m ok > hi.txt
    3. [root@node1 mapreduce]# hadoop fs -put hi.txt /wordcount/input
    4. [root@node1 mapreduce]# hadoop jar hadoop-mapreduce-examples-3.3.6.jar wordcount /wordcount/input /wordcount/output
    5. 2024-01-03 00:19:26,657 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node1/192.168.18.81:8032
    6. 2024-01-03 00:19:27,210 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1704207115702_0002
    7. 2024-01-03 00:19:27,547 INFO input.FileInputFormat: Total input files to process : 1
    8. 2024-01-03 00:19:27,680 INFO mapreduce.JobSubmitter: number of splits:1
    9. 2024-01-03 00:19:27,852 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1704207115702_0002
    10. 2024-01-03 00:19:27,852 INFO mapreduce.JobSubmitter: Executing with tokens: []
    11. 2024-01-03 00:19:28,010 INFO conf.Configuration: resource-types.xml not found
    12. 2024-01-03 00:19:28,010 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
    13. 2024-01-03 00:19:28,072 INFO impl.YarnClientImpl: Submitted application application_1704207115702_0002
    14. 2024-01-03 00:19:28,106 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1704207115702_0002/
    15. 2024-01-03 00:19:28,107 INFO mapreduce.Job: Running job: job_1704207115702_0002
    16. 2024-01-03 00:19:35,223 INFO mapreduce.Job: Job job_1704207115702_0002 running in uber mode : false
    17. 2024-01-03 00:19:35,225 INFO mapreduce.Job:  map 0% reduce 0%
    18. 2024-01-03 00:19:40,342 INFO mapreduce.Job:  map 100% reduce 0%
    19. 2024-01-03 00:19:45,397 INFO mapreduce.Job:  map 100% reduce 100%
    20. 2024-01-03 00:19:45,415 INFO mapreduce.Job: Job job_1704207115702_0002 completed successfully
    21. 2024-01-03 00:19:45,557 INFO mapreduce.Job: Counters: 54
    22.         File System Counters
    23.                 FILE: Number of bytes read=50
    24.                 FILE: Number of bytes written=553581
    25.                 FILE: Number of read operations=0
    26.                 FILE: Number of large read operations=0
    27.                 FILE: Number of write operations=0
    28.                 HDFS: Number of bytes read=119
    29.                 HDFS: Number of bytes written=24
    30.                 HDFS: Number of read operations=8
    31.                 HDFS: Number of large read operations=0
    32.                 HDFS: Number of write operations=2
    33.                 HDFS: Number of bytes read erasure-coded=0
    34.         Job Counters
    35.                 Launched map tasks=1
    36.                 Launched reduce tasks=1
    37.                 Data-local map tasks=1
    38.                 Total time spent by all maps in occupied slots (ms)=2563
    39.                 Total time spent by all reduces in occupied slots (ms)=2582
    40.                 Total time spent by all map tasks (ms)=2563
    41.                 Total time spent by all reduce tasks (ms)=2582
    42.                 Total vcore-milliseconds taken by all map tasks=2563
    43.                 Total vcore-milliseconds taken by all reduce tasks=2582
    44.                 Total megabyte-milliseconds taken by all map tasks=2624512
    45.                 Total megabyte-milliseconds taken by all reduce tasks=2643968
    46.         Map-Reduce Framework
    47.                 Map input records=1
    48.                 Map output records=5
    49.                 Map output bytes=34
    50.                 Map output materialized bytes=50
    51.                 Input split bytes=105
    52.                 Combine input records=5
    53.                 Combine output records=5
    54.                 Reduce input groups=5
    55.                 Reduce shuffle bytes=50
    56.                 Reduce input records=5
    57.                 Reduce output records=5
    58.                 Spilled Records=10
    59.                 Shuffled Maps =1
    60.                 Failed Shuffles=0
    61.                 Merged Map outputs=1
    62.                 GC time elapsed (ms)=118
    63.                 CPU time spent (ms)=1350
    64.                 Physical memory (bytes) snapshot=532602880
    65.                 Virtual memory (bytes) snapshot=5595779072
    66.                 Total committed heap usage (bytes)=392167424
    67.                 Peak Map Physical memory (bytes)=266100736
    68.                 Peak Map Virtual memory (bytes)=2794917888
    69.                 Peak Reduce Physical memory (bytes)=266502144
    70.                 Peak Reduce Virtual memory (bytes)=2800861184
    71.         Shuffle Errors
    72.                 BAD_ID=0
    73.                 CONNECTION=0
    74.                 IO_ERROR=0
    75.                 WRONG_LENGTH=0
    76.                 WRONG_MAP=0
    77.                 WRONG_REDUCE=0
    78.         File Input Format Counters
    79.                 Bytes Read=14
    80.         File Output Format Counters
    81.                 Bytes Written=24
    82. [root@node1 mapreduce]#

    思考：

    1、执行MapReduce的时候，为什么首先请求YARN?

    2、MapReduce看上去好像是两个阶段？先Map，再Reduce？

    3、处理小数据的时候，MapReduce速度快吗？

    Hadoop分布式文件系统(二)

    再小的努力，乘以365都很明显！
    一个程序员最重要的能力是：写出高质量的代码！！
    有道无术，术尚可求也，有术无道，止于术。
    无论你是年轻还是年长，所有程序员都需要记住：时刻努力学习新技术，否则就会被时代抛弃！

  • 相关阅读:
    c#自动生成缺陷图像-添加从待匹配目标文件夹图像及xml移动至指定目标文件中功能--20240612
    配置Kafka消息保留时间
    Grid布局介绍
    论文笔记 - SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios
    ✔ ★ 算法基础笔记（Acwing）（三）—— 搜索与图论（17道题）【java版本】
    jquery中的ajax请求方式的写法 & 重要且常用参数的释义 & ajax返回值，return获取不到数据值
    InfluxDB 数据备份与恢复
    安装深度(Deepin)系统
    Flink SQL --Flink 整合 hive
    九、Spring Boot 缓存（1）
  • 原文地址：https://blog.csdn.net/weixin_42472027/article/details/132957921