Sqoop 安装部署

今天重新部署一台 sqoop，就手再记录一下

官网：sqoop.apache.org

说明：sqoop 有两个大版本，1 和 2，且两个是完全不同的，但 2 现在一直是 1.99.x 版本，并未到 2.x 版本，因此我们选用 1.4.6 版本（当前最新是 1.4.7，使用最新的前一个版本）；

下载、安装

$ tar xzvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /home/hadoop/local/

$ cd /home/hadoop/local

$ ln -s sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop

sqoop 是导入工具，不需要在集群中所有服务器都安装，只在一个服务器安装即可，我装到了 ns1 服务器；

配置

$ cd sqoop/conf

$ mv sqoop-env-template.sh sqoop-env.sh

$ vim sqoop-env.sh

最下面添加：

export HADOOP_COMMON_HOME=/home/local/hadoop                                                                                                                         
export HADOOP_MAPRED_HOME=/home/local/hadoop                                                                                                                         
export HIVE_HOME=/home/hadoop/local/hive                                                                                                                             
export ZOOKEEPER_HOME=/home/hadoop/local/zookeeper                                                                                                                   
export ZOOCFGDIR=/home/hadoop/local/zookeeper/conf

这里配置的 ZOOKEEPER 是使用 HBASE 时候用的

添加环境变量

$ sudo vim /etc/profile.d/my_env.sh

SQOOP_HOME=/home/hadoop/local/sqoop                                                                                                                                  
PATH=$PATH:/home/hadoop/bin:$SQOOP_HOME/bin                      
export SQOOP_HOME PATH    

$ source /etc/profile

拷贝 JDBC 驱动

$ cp mysql-connector-java-8.0.11.jar /home/hadoop/local/sqoop/lib/

验证

$ sqoop version

Warning: /home/hadoop/local/sqoop/../hbase does not exist! HBase imports will fail.                                                                                  
Please set $HBASE_HOME to the root of your HBase installation.                                                                                                       
Warning: /home/hadoop/local/sqoop/../hcatalog does not exist! HCatalog jobs will fail.                                                                               
Please set $HCAT_HOME to the root of your HCatalog installation.                                                                                                     
Warning: /home/hadoop/local/sqoop/../accumulo does not exist! Accumulo imports will fail.                                                                            
Please set $ACCUMULO_HOME to the root of your Accumulo installation.                                                                                                 
2022-09-07 17:14:12,041 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6                                                                                               
Sqoop 1.4.6                                                                                                                                                          
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25                                                                                                               
Compiled by root on Mon Apr 27 14:38:36 CST 2015   

Warning 不用管；

$ sqoop help

usage: sqoop COMMAND [ARGS]                                                                                                                                          
                                                                                                                                                                     
Available commands:                                                                                                                                                  
  codegen            Generate code to interact with database records                                                                                                 
  create-hive-table  Import a table definition into Hive                                                                                                             
  eval               Evaluate a SQL statement and display the results                                                                                                
  export             Export an HDFS directory to a database table                                                                                                    
  help               List available commands                                                                                                                         
  import             Import a table from a database to HDFS                                                                                                          
  import-all-tables  Import tables from a database to HDFS                                                                                                           
  import-mainframe   Import datasets from a mainframe server to HDFS                                                                                                 
  job                Work with saved jobs                                                                                                                            
  list-databases     List available databases on a server                                                                                                            
  list-tables        List available tables in a database                                                                                                             
  merge              Merge results of incremental imports                                                                                                            
  metastore          Run a standalone Sqoop metastore                                                                                                                
  version            Display version information                                                                                                                     
                                                                                                                                                                     
See 'sqoop help COMMAND' for information on a specific command. 

$ sqoop list-databases --connect jdbc:mysql://ns1:3306 --username root --password 123456

mysql                                                                                                                                                                
information_schema                                                                                                                                                   
performance_schema                                                                                                                                                   
sys                                                                                                                                                                  
activiti2                                                                                                                                                            
logistics                                                                                                                                                            
activiti                                                                                                                                                                                                                                                                                                                            
zsoft  

列出了当前 ns1 上面 MySQL 中的所有数据库；