hudi 0.8,使用insert_overwrite_table模式覆寫hudi表蜓萄。之前寫入正常蚁趁,但hive由1.x升級(jí)到2.7.2后干发,報(bào)下面的錯(cuò)誤:
Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL ALTER TABLE `dw_t1`.`contact` REPLACE COLUMNS(`_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `id` bigint, `ts` bigint, `tag_sqlrqx1` bigint, `tag_y_testmulmetadat` double, `tag_sjsmc1` bigint, `tag_qgbq1` string, `tag_sjzb1` double, `tag_sjph1` ARRAY< string>, `tag_sqlbq1` bigint, `tag_sqlsz1` int ) cascade
at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:369)
at org.apache.hudi.hive.HoodieHiveClient.updateTableDefinition(HoodieHiveClient.java:251)
at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:206)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:154)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:108)
... 50 more
Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions :
tag_sqlrqx1,tag_sjsmc1,tag_qgbq1,tag_sjph1
at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:385)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254)
at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:367)
查閱資料后,發(fā)現(xiàn)此問題是hive的問題叙甸,和hudi無關(guān)
問題原因:
而hive.metastore.disallow.incompatible.col.type.changes這個(gè)參數(shù)颖医,在Hive2.0后,默認(rèn)為True裆蒸。即在字段類型轉(zhuǎn)換時(shí)熔萧,不再允許顯式轉(zhuǎn)換(強(qiáng)制轉(zhuǎn)換),而是只能進(jìn)行隱式轉(zhuǎn)換,即由低級(jí)類型轉(zhuǎn)換為高級(jí)類型
解決方法:
在hive服務(wù)端的hive-site.xml文件中添加下面配置佛致,然后重啟hive和hivemetastore服務(wù)
<property>
<name>hive.metastore. disallow.incompatible.col.type.changes</name>
<value>false</value>
</property>
補(bǔ)充:自測(cè)贮缕,在hive客戶端,執(zhí)行set hive.metastore.disallow.incompatible.col.type.changes=false后俺榆,再執(zhí)行REPLACE COLUMNS的sql感昼,還是會(huì)報(bào)錯(cuò)。