1、創(chuàng)建表
建表語(yǔ)法
CREATE [EXTERNAL] TABLE [IF NOT EXISTS] table_name
[(col_name data_type [COMMENT col_comment], ...)]
[COMMENT table_comment]
[PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
[CLUSTERED BY (col_name, col_name, ...)
[SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS]
[ROW FORMAT row_format]
[STORED AS file_format]
[LOCATION hdfs_path]
創(chuàng)建測(cè)試使用的數(shù)據(jù)庫(kù)myhive3,使用該數(shù)據(jù)庫(kù)。
1)帖蔓、創(chuàng)建普通表
0: jdbc:hive2://localhost:10000> create database myhive3;
No rows affected (0.204 seconds)
0: jdbc:hive2://localhost:10000> use myhive3;
No rows affected (0.13 seconds)
0: jdbc:hive2://localhost:10000> create table t1(id int,name string)
0: jdbc:hive2://localhost:10000> row format delimited fields terminated by ',';//指定企软,分割庐扫,具體的參考前面說(shuō)的那篇
No rows affected (0.117 seconds)
0: jdbc:hive2://localhost:10000> show tables ;
+-----------+--+
| tab_name |
+-----------+--+
| t1 |
+-----------+--+
0: jdbc:hive2://localhost:10000> desc t1;
+-----------+------------+----------+--+
| col_name | data_type | comment |
+-----------+------------+----------+--+
| id | int | |
| name | string | |
+-----------+------------+----------+--+
2)、創(chuàng)建外部表
EXTERNAL關(guān)鍵字可以讓用戶(hù)創(chuàng)建一個(gè)外部表,在建表的同時(shí)指定一個(gè)指向?qū)嶋H數(shù)據(jù)的路徑(LOCATION)形庭,Hive 創(chuàng)建內(nèi)部表時(shí)铅辞,會(huì)將數(shù)據(jù)移動(dòng)到數(shù)據(jù)倉(cāng)庫(kù)指向的路徑;若創(chuàng)建外部表萨醒,僅記錄數(shù)據(jù)所在的路徑斟珊,不對(duì)數(shù)據(jù)的位置做任何改變。在刪除表的時(shí)候富纸,內(nèi)部表的元數(shù)據(jù)和數(shù)據(jù)會(huì)被一起刪除囤踩,而外部表只刪除元數(shù)據(jù),不刪除數(shù)據(jù)晓褪。
STORED AS
SEQUENCEFILE|TEXTFILE|RCFILE
如果文件數(shù)據(jù)是純文本堵漱,可以使用 STORED AS TEXTFILE。如果數(shù)據(jù)需要壓縮涣仿,使用 STORED AS SEQUENCEFILE勤庐。
location當(dāng)然是指定表(hdfs上)位置
0: jdbc:hive2://localhost:10000> create external table t2(id int,name string)
0: jdbc:hive2://localhost:10000> row format delimited fields terminated by ','
0: jdbc:hive2://localhost:10000> stored as textfile
0: jdbc:hive2://localhost:10000> location '/mytable2';
No rows affected (0.133 seconds)
頁(yè)面查看是否創(chuàng)建了該表
直接創(chuàng)建在根目錄下的,區(qū)別于普通表創(chuàng)建在/user/hive/warehouse目錄下好港。
3)愉镰、創(chuàng)建分區(qū)
創(chuàng)建分區(qū),分區(qū)字段fields string媚狰,查看表信息的時(shí)候會(huì)顯示該表下所有分區(qū)信息的岛杀。
0: jdbc:hive2://localhost:10000> create table t3(id int,name string)
0: jdbc:hive2://localhost:10000> partitioned by(fields string)
0: jdbc:hive2://localhost:10000> row format delimited fields terminated by ',';
No rows affected (0.164 seconds)
0: jdbc:hive2://localhost:10000> load data local inpath '/root/sz.data' into table t3 partition (fields ='Chengdu');
INFO : Loading data to table myhive3.t3 partition (fields=Chengdu) from file:/root/sz.data
INFO : Partition myhive3.t3{fields=Chengdu} stats: [numFiles=1, numRows=0, totalSize=91, rawDataSize=0]
No rows affected (0.738 seconds)
0: jdbc:hive2://localhost:10000> load data local inpath '/root/sz.data' into table t3 partition (fields ='Wuhan');
INFO : Loading data to table myhive3.t3 partition (fields=Wuhan) from file:/root/sz.data
INFO : Partition myhive3.t3{fields=Wuhan} stats: [numFiles=1, numRows=0, totalSize=91, rawDataSize=0]
No rows affected (0.608 seconds)
0: jdbc:hive2://localhost:10000> select * from t3;
+--------+-----------+------------+--+
| t3.id | t3.name | t3.fields |
+--------+-----------+------------+--+
| 1 | zhangsan | Chengdu |
| 2 | lisi | Chengdu |
| 3 | wangwu | Chengdu |
| 4 | furong | Chengdu |
| 5 | fengjie | Chengdu |
| 6 | aaa | Chengdu |
| 7 | bbb | Chengdu |
| 8 | ccc | Chengdu |
| 9 | ddd | Chengdu |
| 10 | eee | Chengdu |
| 11 | fff | Chengdu |
| 12 | ggg | Chengdu |
| 1 | zhangsan | Wuhan |
| 2 | lisi | Wuhan |
| 3 | wangwu | Wuhan |
| 4 | furong | Wuhan |
| 5 | fengjie | Wuhan |
| 6 | aaa | Wuhan |
| 7 | bbb | Wuhan |
| 8 | ccc | Wuhan |
| 9 | ddd | Wuhan |
| 10 | eee | Wuhan |
| 11 | fff | Wuhan |
| 12 | ggg | Wuhan |
+--------+-----------+------------+--+
頁(yè)面查看
這兩個(gè)分區(qū)目錄下都存放了文件sz.data。
2崭孤、修改表
1)类嗤、增加、刪除表分區(qū)
語(yǔ)法
增加
ALTER TABLE table_name ADD [IF NOT EXISTS] partition_spec [ LOCATION 'location1' ] partition_spec [ LOCATION 'location2' ] ...
刪除
ALTER TABLE table_name DROP partition_spec, partition_spec,...
還是對(duì)上面的分區(qū)表t3
增加分區(qū)fields=’Hefei’位置還是跟其他分區(qū)一致(可以省略不寫(xiě))
由于hive客戶(hù)端命令行可以使用hadoop命令查看文件系統(tǒng)(dfs)辨宠,后面就不去頁(yè)面查看了
0: jdbc:hive2://localhost:10000> alter table t3 add partition (fields='Hefei');
No rows affected (0.198 seconds)
0: jdbc:hive2://localhost:10000> dfs -ls /user/hive/warehouse/myhive3.db/t3;
+---------------------------------------------------------------------------------------------------------------+--+
| DFS Output |
+---------------------------------------------------------------------------------------------------------------+--+
| Found 3 items |
| drwxr-xr-x - root supergroup 0 2017-10-19 05:17 /user/hive/warehouse/myhive3.db/t3/fields=Chengdu |
| drwxr-xr-x - root supergroup 0 2017-10-19 05:28 /user/hive/warehouse/myhive3.db/t3/fields=Hefei |
| drwxr-xr-x - root supergroup 0 2017-10-19 05:18 /user/hive/warehouse/myhive3.db/t3/fields=Wuhan |
+---------------------------------------------------------------------------------------------------------------+--+
0: jdbc:hive2://localhost:10000> alter table t3 drop partition (fields='Hefei');
INFO : Dropped the partition fields=Hefei
No rows affected (0.536 seconds)
0: jdbc:hive2://localhost:10000> dfs -ls /user/hive/warehouse/myhive3.db/t3;
+---------------------------------------------------------------------------------------------------------------+--+
| DFS Output |
+---------------------------------------------------------------------------------------------------------------+--+
| Found 2 items |
| drwxr-xr-x - root supergroup 0 2017-10-19 05:17 /user/hive/warehouse/myhive3.db/t3/fields=Chengdu |
| drwxr-xr-x - root supergroup 0 2017-10-19 05:18 /user/hive/warehouse/myhive3.db/t3/fields=Wuhan |
+---------------------------------------------------------------------------------------------------------------+--+
2)遗锣、重命名表
語(yǔ)法
alter table old_name rename to new_name
將t1改名為t4
0: jdbc:hive2://localhost:10000> alter table t1 rename to t4;
No rows affected (0.183 seconds)
0: jdbc:hive2://localhost:10000> show tables;
+-----------+--+
| tab_name |
+-----------+--+
| t2 |
| t3 |
| t4 |
+-----------+--+
3 rows selected (0.127 seconds)
3)、添加嗤形、更新列
語(yǔ)法
alter table table_name add|replace columns(col_name data_type ...)
注:ADD是代表新增一字段精偿,字段位置在所有列后面,REPLACE則是表示替換表中所有字段赋兵。
0: jdbc:hive2://localhost:10000> desc t4;
+-----------+------------+----------+--+
| col_name | data_type | comment |
+-----------+------------+----------+--+
| id | int | |
| name | string | |
+-----------+------------+----------+--+
2 rows selected (0.315 seconds)
0: jdbc:hive2://localhost:10000> alter table t4 add columns (age int);
No rows affected (0.271 seconds)
0: jdbc:hive2://localhost:10000> desc t4;
+-----------+------------+----------+--+
| col_name | data_type | comment |
+-----------+------------+----------+--+
| id | int | |
| name | string | |
| age | int | |
+-----------+------------+----------+--+
3 rows selected (0.199 seconds)
0: jdbc:hive2://localhost:10000> alter table t4 replace columns (no string,name string,scores int);
No rows affected (0.406 seconds)
0: jdbc:hive2://localhost:10000> desc t4;
+-----------+------------+----------+--+
| col_name | data_type | comment |
+-----------+------------+----------+--+
| no | string | |
| name | string | |
| scores | int | |
+-----------+------------+----------+--+
常用顯示命令
show tables
show databases
show partitions
show functions
desc formatted table_name;//跟desc table_name一樣笔咽,但是顯示的內(nèi)容更多
3、數(shù)據(jù)操作
1)霹期、load導(dǎo)入數(shù)據(jù)
上面已經(jīng)演示了將本地的文件sz.data導(dǎo)入到t3表中叶组。
load也就是說(shuō)將文件復(fù)制到指定的表(目錄)下,指定了local的話那么會(huì)去查找本地文件系統(tǒng)中的文件路徑历造。如果沒(méi)指定會(huì)根據(jù)inpath指定的路徑去查找甩十。如果是hdfs的話船庇,如下格式
hdfs://namenode:9000/user/hive/project/data1。
另外如果使用了 OVERWRITE 關(guān)鍵字侣监,則目標(biāo)表(或者分區(qū))中的內(nèi)容會(huì)被刪除鸭轮,然后再將 filepath 指向的文件/目錄中的內(nèi)容添加到表/分區(qū)中。
如果目標(biāo)表(分區(qū))已經(jīng)有一個(gè)文件橄霉,并且文件名和 filepath 中的文件名沖突窃爷,那么現(xiàn)有的文件會(huì)被新文件所替代。
0: jdbc:hive2://localhost:10000> load data local inpath '/root/sz.data' overwrite into table t4 ;
INFO : Loading data to table myhive3.t4 from file:/root/sz.data
INFO : Table myhive3.t4 stats: [numFiles=1, numRows=0, totalSize=91, rawDataSize=0]
No rows affected (0.7 seconds)
0: jdbc:hive2://localhost:10000> select * from t4;
+--------+-----------+------------+--+
| t4.no | t4.name | t4.scores |
+--------+-----------+------------+--+
| 1 | zhangsan | NULL |
| 2 | lisi | NULL |
| 3 | wangwu | NULL |
| 4 | furong | NULL |
| 5 | fengjie | NULL |
| 6 | aaa | NULL |
| 7 | bbb | NULL |
| 8 | ccc | NULL |
| 9 | ddd | NULL |
| 10 | eee | NULL |
| 11 | fff | NULL |
| 12 | ggg | NULL |
+--------+-----------+------------+--+
2)酪劫、插入語(yǔ)句
向表中插入語(yǔ)句的話
普通插入吞鸭,查詢(xún)其他表的表信息插入(自動(dòng)數(shù)量要一致),將查詢(xún)結(jié)果保存到一個(gè)目錄中(目錄會(huì)自動(dòng)創(chuàng)建覆糟,由OutputFormat實(shí)現(xiàn))刻剥。
insert into table t4 values('13','zhangsan',99);
0: jdbc:hive2://localhost:10000> truncate table t4;//清空表信息
0: jdbc:hive2://localhost:10000> insert into t4
0: jdbc:hive2://localhost:10000> select id,name from t3;
0: jdbc:hive2://localhost:10000> select * from t4;
+--------+-----------+--+
| t4.no | t4.name |
+--------+-----------+--+
| 1 | zhangsan |
| 2 | lisi |
| 3 | wangwu |
| 4 | furong |
| 5 | fengjie |
| 6 | aaa |
| 7 | bbb |
| 8 | ccc |
| 9 | ddd |
| 10 | eee |
| 11 | fff |
| 12 | ggg |
| 1 | zhangsan |
| 2 | lisi |
| 3 | wangwu |
| 4 | furong |
| 5 | fengjie |
| 6 | aaa |
| 7 | bbb |
| 8 | ccc |
| 9 | ddd |
| 10 | eee |
| 11 | fff |
| 12 | ggg |
+--------+-----------+--+
重新創(chuàng)建表t5,將表信息保存到本地目錄/root/insertDir/test中
0: jdbc:hive2://localhost:10000> insert overwrite local directory '/root/insertDir/test'
0: jdbc:hive2://localhost:10000> select * from t5;
查看本地
[root@mini1 ~]# cd insertDir/test/
[root@mini1 test]# ll
總用量 4
-rw-r--r--. 1 root root 91 10月 19 06:15 000000_0
[root@mini1 test]# cat 000000_0
1zhangsan
2lisi
3wangwu
4furong
5fengjie
6aaa
7bbb
8ccc
9ddd
10eee
11fff
12ggg
4滩字、數(shù)據(jù)查詢(xún)SELECT
語(yǔ)法基本跟mysql一樣造虏,留意下分桶即可
SELECT [ALL | DISTINCT] select_expr, select_expr, ...
FROM table_reference
[WHERE where_condition]
[GROUP BY col_list [HAVING condition]]
[CLUSTER BY col_list
| [DISTRIBUTE BY col_list] [SORT BY| ORDER BY col_list]
]
[LIMIT number]
在前面做了很多測(cè)試,就不想再重復(fù)了麦箍,會(huì)mysql的查詢(xún)這個(gè)肯定也會(huì)漓藕。
需要注意的是order by和sort by的區(qū)別:
1、order by 會(huì)對(duì)輸入做全局排序挟裂,因此只有一個(gè)reducer享钞,會(huì)導(dǎo)致當(dāng)輸入規(guī)模較大時(shí),需要較長(zhǎng)的計(jì)算時(shí)間诀蓉。
2栗竖、sort by不是全局排序,其在數(shù)據(jù)進(jìn)入reducer前完成排序渠啤。因此狐肢,如果用sort by進(jìn)行排序,并且設(shè)置mapred.reduce.tasks>1沥曹,則sort by只保證每個(gè)reducer的輸出有序份名,不保證全局有序。
主要介紹下join
5妓美、Join查詢(xún)
join查詢(xún)其實(shí)跟mysql還是一樣的
準(zhǔn)備數(shù)據(jù)
a.txt中
1,a
2,b
3,c
4,d
7,y
8,u
b.txt中
2,bb
3,cc
7,yy
9,pp
創(chuàng)建表a和b僵腺,將a.txt導(dǎo)入到a表中,b.txt導(dǎo)入到b表中
1)壶栋、內(nèi)連接
0: jdbc:hive2://localhost:10000> create table a(id int,name string)
0: jdbc:hive2://localhost:10000> row format delimited fields terminated by ',';
No rows affected (0.19 seconds)
0: jdbc:hive2://localhost:10000> create table b(id int,name string)
0: jdbc:hive2://localhost:10000> row format delimited fields terminated by ',';
No rows affected (0.071 seconds)
0: jdbc:hive2://localhost:10000> load data local inpath '/root/a.txt' into table a;
0: jdbc:hive2://localhost:10000> load data local inpath '/root/b.txt' into table b;
0: jdbc:hive2://localhost:10000> select * from a;
+-------+---------+--+
| a.id | a.name |
+-------+---------+--+
| 1 | a |
| 2 | b |
| 3 | c |
| 4 | d |
| 7 | y |
| 8 | u |
+-------+---------+--+
6 rows selected (0.218 seconds)
0: jdbc:hive2://localhost:10000> select * from b;
+-------+---------+--+
| b.id | b.name |
+-------+---------+--+
| 2 | bb |
| 3 | cc |
| 7 | yy |
| 9 | pp |
+-------+---------+--+
4 rows selected (0.221 seconds)
0: jdbc:hive2://localhost:10000> select * from a inner join b on a.id = b.id;
...
+-------+---------+-------+---------+--+
| a.id | a.name | b.id | b.name |
+-------+---------+-------+---------+--+
| 2 | b | 2 | bb |
| 3 | c | 3 | cc |
| 7 | y | 7 | yy |
+-------+---------+-------+---------+--+
根據(jù)id進(jìn)行連接想邦,能連接到的則串起來(lái)。
2)委刘、左外連接(outer可噬ッ弧)
0: jdbc:hive2://localhost:10000> select * from a left outer join b on a.id = b.id;
...
+-------+---------+-------+---------+--+
| a.id | a.name | b.id | b.name |
+-------+---------+-------+---------+--+
| 1 | a | NULL | NULL |
| 2 | b | 2 | bb |
| 3 | c | 3 | cc |
| 4 | d | NULL | NULL |
| 7 | y | 7 | yy |
| 8 | u | NULL | NULL |
+-------+---------+-------+---------+--+
6 rows selected (16.453 seconds)
左邊的表內(nèi)容全列出來(lái),右邊的能連上的就顯示锡移,不能的則顯示null呕童。
右外連接則相反。
3)淆珊、全連接full outer
0: jdbc:hive2://localhost:10000> select * from a full outer join b on a.id = b.id;
...
+-------+---------+-------+---------+--+
| a.id | a.name | b.id | b.name |
+-------+---------+-------+---------+--+
| 1 | a | NULL | NULL |
| 2 | b | 2 | bb |
| 3 | c | 3 | cc |
| 4 | d | NULL | NULL |
| 7 | y | 7 | yy |
| 8 | u | NULL | NULL |
| NULL | NULL | 9 | pp |
+-------+---------+-------+---------+--+
相當(dāng)于左連接+右連接
4)夺饲、semi join
0: jdbc:hive2://localhost:10000> select * from a left semi join b on a.id = b.id;
+-------+---------+--+
| a.id | a.name |
+-------+---------+--+
| 2 | b |
| 3 | c |
| 7 | y |
+-------+---------+--+
3 rows selected (17.511 seconds)
相當(dāng)于左外連接得到的信息的左半部分。
注:可以理解為exist in(…)施符,但是hive中沒(méi)有該語(yǔ)法往声,所以使用LEFT SEMI JOIN代替IN/EXISTS的,前者為后者高效實(shí)現(xiàn)戳吝。
比如下面的例子
重寫(xiě)以下子查詢(xún)?yōu)長(zhǎng)EFT SEMI JOIN
SELECT a.key, a.value
FROM a
WHERE a.key exist in
(SELECT b.key
FROM B);
可以被重寫(xiě)為:
SELECT a.key, a.val
FROM a LEFT SEMI JOIN b on (a.key = b.key)