Hadoop(四)

Hadoop部署

一、配置core-site.xml。代理主机

二、HDFS配置文件

三、YARN配置

Hadoop部署

一、配置core-site.xml。代理主机

[doudou@hadoop101 module]$ cd hadoop-3.1.3/

[doudou@hadoop101 hadoop-3.1.3]$ cd etc/hadoop/

[doudou@hadoop101 hadoop]$ cd core-site.xml

[doudou@hadoop101 hadoop]$ vim /core-site.xml


<configuration>
  
  <property>
     <name>fs.defaultFSname>
     <value>hdfs://hadoop102:8020value>
  property>
  <property>
     <name>hadoop.tmp.dirname>
     <value>/opt/module/hadoop-3.1.3/datavalue>
  property>
  <property>
     <name>hadoop.http.staticuser.username>
     <value>doudouvalue>
  property>
  <property>
       <name>hadoop.proxyuser.doudou.hostsname>
       <value>*value>
  property>
  <property>
     <name>hadoop.proxyuser.doudou.grupsname>
     <value>*value>
  property>
  <property>
       <name>hadoop.proxyuser.doudou.usersname>
         <value>*value>
  property>
 
configuration>

二、HDFS配置文件

配置hdfs-site.xml

[doudou@hadoop101 hadoop]$ vim hdfs-site.xml


<configuration>
  <property>
   <name>dfs.namenode.http-addressname>
   <value>hadoop101:9870value>
  property>
  <property>
    <name>dfs.namenode.secondary.http-addressname>
    <value>hadoop103:9868value>
  property>
  <property>
     <name>dfs.replicationname>
     <value>1value>
  property>
 
configuration>
~

分发

[doudou@hadoop101 hadoop]$ xsync hdfs-site.xml

三、YARN配置

配置yarn-site.xml

[doudou@hadoop101 hadoop]$ vim yarn-site.xml


<configuration>
 
   
    <property>
        <name>yarn.nodemanager.aux-servicesname>
        <value>mapreduce_shufflevalue>
    property>
 
    
    <property>
        <name>yarn.resourcemanager.hostnamename>
        <value>hadoop103value>
    property>
 
    
    <property>
        <name>yarn.nodemanager.env-whitelistname>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOMEvalue>
    property>
   <property>
       <name>yarn.scheduler.minimum-allocation-mbname>
       <value>512value>
   property>
   <property>
       <name>yarn.scheduler.maximum-allocation-mbname>
       <value>4096value>
   property>
   <property>
        <name>yarn.nodemanager.resource.memory-mbname>
        <value>4096value>
   property>
   <property>
      <name>yarn.nodemanager.vmem-check-enabledname>
      <value>falsevalue>
   property>
configuration>

[doudou@hadoop101 hadoop]$ xsync yarn-site.xml

四、MapReduce配置文件

配置mapred-site.xml


[doudou@hadoop101 hadoop]$ vim mapred-site.xml


<configuration>
	
    <property>
        <name>mapreduce.framework.namename>
        <value>yarnvalue>
    property>
configuration>

[doudou@hadoop101 hadoop]$ xsync mapred-site.xml

五、配置workers

[doudou@hadoop101 ~]$ vim /opt/module/hadoop-3.1.3/etc/hadoop/workers


hadoop101
hadoop102
hadoop103

[doudou@hadoop101 hadoop]$ xsync workers

配置历史服务器

一、配置mapred-site.xml

[doudou@hadoop101 hadoop]$ vim mapred-site.xml


<property>
    <name>mapreduce.jobhistory.webapp.addressname>
    <value>hadoop101:19888value>
property>
>mapreduce.framework.namename>
        <value>yarnvalue>
    property>

[doudou@hadoop101 hadoop]$ xsync mapred-site.xml

配置日志的聚集

一、配置yarn-site.xml

[doudou@hadoop101 hadoop]$ vim yarn-site.xml


<property>
    <name>yarn.log-aggregation-enablename>
    <value>truevalue>
property>
<property>  
    <name>yarn.log.server.urlname>  
    <value>http://hadoop101:19888/jobhistory/logsvalue>
property>
<property>
    <name>yarn.log-aggregation.retain-secondsname>
    <value>604800value>
property>

[doudou@hadoop101 hadoop]$ xsync yarn-site.xml

相关阅读:
广度优先搜索(Breadth First Search, BFS)算法
总估值接近400亿！华秋电子入选阿里云智能制造加速器
DataScience&ML：基于心脏病分类预测数据集利用等算法实现模型可解释性之详细攻略
HTTP状态 400 - 错误的请求：在请求目标中找到无效字符。有效字符在RFC 7230和RFC 3986中定义（Tomcat，非SpringBoot）
新突破！德国MPQ证实光子是量子比特载波的优选
21-30==c++知识点
【AI视野·今日Robot 机器人论文速览第五十六期】Tue, 17 Oct 2023
Java中利用OpenCV进行人脸识别
线段树维护矩阵：0920T4
DSP的PWM

原文地址：https://blog.csdn.net/qq_61741696/article/details/126292707