sqoop可以采集关系型数据库中的数据到hdfs 等
官网链接 https://sqoop.apache.org/
官网使用链接 https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html

一、下载压缩包

下载链接 http://archive.apache.org/dist/sqoop/


 放在任意位置，解压到 /usr/local目录下
 在压缩包的目录下输入 tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /usr/local

 重命名

[root@node3 tmp]# cd /usr/local
[root@node3 local]# mv sqoop-1.4.7.bin__hadoop-2.6.0/ sqoop

1
2
3

 sqoop的安装目录

[root@node3 sqoop]# pwd
/usr/local/sqoop
[root@node3 sqoop]# 
1
2
3

二、配置环境变量

vi /etc/profile
1

SQOOP_HOME=/usr/local/sqoop
export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$FLUME_HOME/bin:$CATALINA_HOME/bin:$SQOOP_HOME/bin
export HADOOP_HOME ZOOKEEPER_HOME FLUME_HOME CATALINA_HOME SQOOP_HOME

1
2
3
4

source /etc/profile
1

三、其他配置

 记住hadoop的安装目录

[root@node3 sqoop]# echo $HADOOP_HOME
/usr/local/hadoop271

1
2
3
4

 向sqoop配置文件中添加hadoop路径

 vi $SQOOP_HOME/conf/sqoop-env.sh

1
2

export HADOOP_HOME=/usr/local/hadoop271

1
2

 向sqoop目录下的lib加入mysql的驱动包

[root@node3 sqoop]# cd $SQOOP_HOME/lib
[root@node3 lib]# ls

1
2
3

 向hadoop目录下的lib加入sqoop的jar包

[root@node3 hadoop271]# cd $HADOOP_HOME/lib
[root@node3 lib]# ls
native  sqoop-1.4.7.jar

1
2
3
4

 寻找jar包的链接 https://mvnrepository.com/
 sqoop1.4.7的jar包 https://mvnrepository.com/artifact/org.apache.sqoop/sqoop/1.4.7
 mysql8的驱动包
https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.28

四、测试

 查看sqoop版本：

[root@node3 lib]# sqoop version
Warning: /usr/local/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
22/06/28 19:19:15 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017
[root@node3 lib]# 

1
2
3
4
5
6
7
8
9

 查看数据库:

 sqoop list-databases --connect jdbc:mysql://localhost:3306/mysql?serverTimezone=UTC  --username root --password mysql的密码 --verbose

1
2

 导入数据库中的某张表:

sqoop import --connect jdbc:mysql://node3:3306/test?serverTimezone=UTC --username root --password mysql的密码 --table 表名


1
2
3