Sqoop软件安装使用(1.4.5) 2015-02-27 16:00

说明

Sqoop是一个客户端软件,安装在任意机器上,不需要安装在Hadoop集群中。

安装过程

安装

cd /opt/hadoop/client/
wget http://mirror.bit.edu.cn/apache/sqoop/1.4.5/sqoop-1.4.5.bin__hadoop-2.0.4-alpha.tar.gz
tar -zxvf sqoop-1.4.5.bin__hadoop-2.0.4-alpha.tar.gz
mv sqoop-1.4.5.bin__hadoop-2.0.4-alpha sqoop

环境变量

在/etc/profile中增加环境变量:

1
2
export SQOOP_HOME=/opt/hadoop/client/sqoop
export PATH=$SQOOP_HOME/bin:$PATH

配置文件:

创建配置文件:

1
cp sqoop-env-template.sh sqoop-env.sh

修改配置文件:

1
2
3
4
vi sqoop-env.sh
export HADOOP_COMMON_HOME=/opt/hadoop/client/hadoop-2.4.1
export HADOOP_MAPRED_HOME=/opt/hadoop/client/hadoop-2.4.1
export HIVE_HOME=/opt/hadoop/client/hive

MySQL JDBC Jar包

将MySQL的JDBC Jar包拷贝到Sqoop的lib目录下:

1
cp ~/mysql-connector-java-5.1.22-bin.jar /opt/hadoop/client/sqoop/lib/

运行

将MySQL的数据库szhouse中的house_project表导入HDFS中:

sqoop import --connect jdbc:mysql://10.255.1.6/LemonHouse --username root --password 123456 --table house_project

默认设置下导入到HDFS上的路径是: /user/{username}/{tablename}/(files),比如我的当前用户是hadoop,那么实际路径即: /user/hadoop/house_project/(files)。

[hadoop@ctrl ~]$ hdfs dfs -ls /user/hadoop/house_project
Found 5 items
-rw-r--r--   3 hadoop supergroup          0 2015-02-27 16:00 /user/hadoop/house_project/_SUCCESS
-rw-r--r--   3 hadoop supergroup       7972 2015-02-27 16:00 /user/hadoop/house_project/part-m-00000
-rw-r--r--   3 hadoop supergroup      11074 2015-02-27 16:00 /user/hadoop/house_project/part-m-00001
-rw-r--r--   3 hadoop supergroup      13917 2015-02-27 16:00 /user/hadoop/house_project/part-m-00002
-rw-r--r--   3 hadoop supergroup      14020 2015-02-27 16:00 /user/hadoop/house_project/part-m-00003

如果要自定义路径需要增加参数:–warehouse-dir 比如:

sqoop import --connect jdbc:mysql://10.255.1.6/LemonHouse --username root --password 123456 --table house_project --warehouse-dir /tmp/data
Tags: #Sqoop    Post on ETL