• 手把手教你datart集成hive3.1.2


    datart集成hive3.1.2

    datart中是没有hive驱动包的,需要我们自己根据hive版本,放入驱动包

    1.放入hive的驱动jar包

    cp hive-jdbc-3.1.2.jar /opt/datart/lib/
    

    测试hive数据源连接,报如下 错误信息

    2022-09-28 13:34:33.756 ERROR datart.server.job.SchemaSyncJob : source schema sync error 
    com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 5000, active 0, maxActive 8, creating 0, createErrorCount 2
            at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1736)
            at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1408)
            at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1388)
            at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1378)
            at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:99)
            at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.getConn(JdbcDataProviderAdapter.java:277)
            at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.readAllDatabases(JdbcDataProviderAdapter.java:114)
            at datart.data.provider.JdbcDataProvider.readAllDatabases(JdbcDataProvider.java:73)
            at datart.data.provider.ProviderManager.readAllDatabases(ProviderManager.java:87)
            at datart.server.service.impl.DataProviderServiceImpl.readAllDatabases(DataProviderServiceImpl.java:127)
            at datart.server.job.SchemaSyncJob.execute(SchemaSyncJob.java:81)
            at datart.server.job.SchemaSyncJob.execute(SchemaSyncJob.java:72)
            at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
            at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
    Caused by: java.lang.NoClassDefFoundError: org/apache/thrift/transport/TTransport
            at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
            at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1652)
            at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1718)
            at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2785)
    Caused by: java.lang.ClassNotFoundException: org.apache.thrift.transport.TTransport
            at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
            at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
            ... 4 common frames omitted
    

    2.放hive相关jar包

    光有驱动包是不够的,继续放入相关的jar包

     hive-common-3.1.2.jar
     hive-exec-3.1.2.jar
     hive-jdbc-3.1.2.jar
     hive-metastore-3.1.2.jar
     hive-serde-3.1.2.jar
     hive-service-3.1.2.jar
     hive-shims-0.23-3.1.2.jar
     hive-shims-3.1.2.jar
     hive-shims-common-3.1.2.jar
     hive-shims-scheduler-3.1.2.jar
    

    将以上jar包放入datart目录后,包如下错误:

    2022-09-28 13:47:08.907 WARN  com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
    2022-09-28 13:47:13.909 ERROR datart.server.config.WebExceptionHandler : wait millis 5000, active 0, maxActive 8, creating 0, createErrorCount 2
    com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 5000, active 0, maxActive 8, creating 0, createErrorCount 2
            at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1736)
            at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1408)
            at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1388)
            at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1378)
            at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:99)
            at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.getConn(JdbcDataProviderAdapter.java:277)
            at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.execute(JdbcDataProviderAdapter.java:223)
            at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.executeOnSource(JdbcDataProviderAdapter.java:459)
            at datart.data.provider.JdbcDataProvider.execute(JdbcDataProvider.java:95)
            at datart.data.provider.ProviderManager.run(ProviderManager.java:238)
            at datart.data.provider.ProviderManager.execute(ProviderManager.java:123)
            at datart.server.service.impl.DataProviderServiceImpl.testExecute(DataProviderServiceImpl.java:209)
            at datart.server.controller.DataProviderController.testExecute(DataProviderController.java:95)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
            at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
            at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894)
            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
            at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
            at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060)
            at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962)
            at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
            at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
            at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
            at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
            at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
            at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
            at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
            at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
            at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
            at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
            at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
            at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
            at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
            at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
            at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
            at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
            at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
            at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
            at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
            at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
            at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
            at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
            at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
            at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
            at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:887)
            at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1684)
            at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
            at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.NoClassDefFoundError: org/apache/curator/RetryPolicy
            at org.apache.hive.jdbc.Utils.extractURLComponents(Utils.java:472)
            at org.apache.hive.jdbc.Utils.parseURL(Utils.java:332)
            at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:168)
            at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
            at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1652)
            at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1718)
            at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2785)
    Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
            at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
            at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
            ... 7 common frames omitted
    
    

    3.放入curator相关jar包

    查看hive中curator的包

    ll | grep curator
    #有如下相关的jar
    curator-client-2.12.0.jar
    curator-framework-2.12.0.jar
    curator-recipes-2.12.0.jar
    

    放入datart的lib目录下

    cp curator* /opt/datart/lib/
    

    再测试连接,大功告成!

  • 相关阅读:
    202310-MetaPhlAn4安装和使用方法-Anaconda3- centos9 stream
    网页设计学习记录-常用圆角按钮css
    宏观视角下的浏览器
    国密gmtls协议-双证书体系的服务端和客户端通信代码
    flutter系列之:flutter中常用的GridView layout详解
    速通已经“逝去“的jQuery-笔记打卡
    建筑能源管理(5)——建筑能源审计和审计方法
    VS Code + Remote-ssh插件实现远程开发
    FIX:Foxit PDF SDK ActiveX 5.9.9 Crack
    软件工程师必读的13本书
  • 原文地址:https://blog.csdn.net/weixin_52918377/article/details/127088995