• spark 集群启动出现 Unrecognized Hadoop major version number: 3.0.0-cdh6.3.2

    我是cdh集群，如果你不是cdh6.3以上，你可以走了

    这是我的pom，我现在在本地往hive写数据，没问题，但是，如果打成jar包，放到服务器上，执行就有问题

    2. <project xmlns="http://maven.apache.org/POM/4.0.0"
    3.          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    4.          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    5.     <modelVersion>4.0.0modelVersion>
    6.     <properties>
    7.         <project.build.sourceEncoding>UTF-8project.build.sourceEncoding>
    8.         <project.reporting.outputEncoding>UTF-8project.reporting.outputEncoding>
    9.         <maven.compiler.source>1.8maven.compiler.source>
    10.         <maven.compiler.target>1.8maven.compiler.target>
    11.  
    12.         <scala.version>2.11.12scala.version>
    13.         <spark.version>2.4.0spark.version>
    14.         <java.version>1.8java.version>
    15.  
    16.     properties>
    17.     <groupId>org.examplegroupId>
    18.     <artifactId>DataPrepareartifactId>
    19.     <version>1.0-SNAPSHOTversion>
    20.     <dependencies>
    21.         
    22.         <dependency>
    23.             <groupId>org.scala-langgroupId>
    24.             <artifactId>scala-libraryartifactId>
    25.             <version>${scala.version}version>
    26.         dependency>
    27.     <dependency>
    28.         <groupId>org.apache.sparkgroupId>
    29.         <artifactId>spark-core_2.11artifactId>
    30.         <version>${spark.version}version>
    31.     dependency>
    32.         <dependency>
    33.             <groupId>org.apache.sparkgroupId>
    34.             <artifactId>spark-hive_2.11artifactId>
    35.             <version>${spark.version}version>
    36.         dependency>
    37.     <dependency>
    38.         <groupId>org.apache.sparkgroupId>
    39.         <artifactId>spark-sql_2.11artifactId>
    40.         <version>${spark.version}version>
    41.     dependency>
    42.  
    43.     dependencies>
    44.    
    45.     <build>
    46.         <finalName>WordCountfinalName>
    47.         <plugins>
    48.             <plugin>
    49.                 <groupId>net.alchim31.mavengroupId>
    50.                 <artifactId>scala-maven-pluginartifactId>
    51.                 <version>3.2.2version>
    52.                 <executions>
    53.                     <execution>
    54.                         <goals>
    55.                             <goal>compilegoal>
    56.                             <goal>testCompilegoal>
    57.                         goals>
    58.                     execution>
    59.                 executions>
    60.             plugin>
    61.             <plugin>
    62.                 <groupId>org.apache.maven.pluginsgroupId>
    63.                 <artifactId>maven-assembly-pluginartifactId>
    64.                 <version>3.0.0version>
    65.                 <configuration>
    66.                     <archive>
    67.                         <manifest>
    68.                             <mainClass>WordCountmainClass>
    69.                         manifest>
    70.                     archive>
    71.                     <descriptorRefs>
    72.                         <descriptorRef>jar-with-dependenciesdescriptorRef>
    73.                     descriptorRefs>
    74.                 configuration>
    75.                 <executions>
    76.                     <execution>
    77.                         <id>make-assemblyid>
    78.                         <phase>packagephase>
    79.                         <goals>
    80.                             <goal>singlegoal>
    81.                         goals>
    82.                     execution>
    83.                 executions>
    84.             plugin>
    85.         plugins>
    86.     build>
    87.  
    88. project>

     

    后来了解到，只有在用代码集成hive-jdbc，spark-core，spark-sql，spark-hive等依赖的时候，就会报出这么一个问题。

    spark-hive.jar包和spark2.4有冲突，这个是一个社区问题，不是你的问题。

    本地，idea运行时，需要有这个spark-hive，但是你上传到集群交给，yarn去执行每台机器的spark就必须把他注释掉。

    解决方案：丢到集群的时候，注释掉，在进行打包

    2.     
    3.     
    4.     
    5.