今天是2021年最后一天临谱,提前祝大家元旦快樂,這也是2021年最后一篇文章奴璃,后續(xù)會(huì)給大家?guī)砀嚓P(guān)于Doris的文章悉默,同時(shí)也希望Doris 2022年起飛,順利從Apache 孵化器畢業(yè)成頂級(jí)項(xiàng)目苟穆,給大家?guī)砀斐巍⒏€(wěn)定、生態(tài)更豐富的MPP OLAP分析型數(shù)據(jù)庫產(chǎn)品雳旅。
Apache Doris 社區(qū) 2022 年的總體規(guī)劃跟磨,包括待開展或已開展、以及已完成但需要持續(xù)優(yōu)化的功能攒盈、文檔抵拘、社區(qū)建設(shè)等多方面,我們期待有更多的小伙伴參與進(jìn)來討論型豁。同時(shí)也希望多多關(guān)注Doris僵蛛,給Doris加Star
之前寫了在Centos下Apache Doris 外表使用方法及注意實(shí)現(xiàn)尚蝌,但是不少用戶是使用ubuntu系統(tǒng),在這個(gè)系統(tǒng)下很多用戶遇到ODBC外表導(dǎo)致BE服務(wù)宕機(jī)的問題墩瞳,對(duì)此我專門在ubuntu 18.04下進(jìn)行了測試
1.軟件環(huán)境
- 操作系統(tǒng):ubuntu 18.04
- Apache Doris :0.15
- mysql 數(shù)據(jù)庫:8.0.27-0ubuntu0.20.04.1 for Linux on x86_64
- UnixODBC:2.3.4
- Mysql Connector ODBC :5.3.13驼壶、8.0.11、8.0.26
2.安裝ODBC驅(qū)動(dòng)
首先我們安裝unixODBC驅(qū)動(dòng)喉酌、這里直接給出驅(qū)動(dòng)的下載地址及安裝命令
sudo wget ftp://ftp.unixodbc.org/pub/unixODBC/unixODBC-2.3.4.tar.gz
tar -xvzf unixODBC-2.3.4.tar.gz
cd unixODBC-2.3.4/
sudo ./configure --prefix=/usr/local/unixODBC-2.3.7 --includedir=/usr/include --libdir=/usr/lib -bindir=/usr/bin --sysconfdir=/etc
make
sudo make install
安裝成功后热凹,unixODBC所需的頭文件都被安裝到了/usr/inlucde下,編譯好的庫文件安裝到了/usr/lib下泪电,與unixODBC相關(guān)的可執(zhí)行文件安裝到了/usr/bin下般妙,配置文件放到了/etc下。
驗(yàn)證安裝是否成功
# odbcinst -j
unixODBC 2.3.4
DRIVERS............: /etc/odbcinst.ini
SYSTEM DATA SOURCES: /etc/odbc.ini
FILE DATA SOURCES..: /etc/ODBCDataSources
USER DATA SOURCES..: /root/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
3.安裝Mysql ODBC驅(qū)動(dòng)
這里我默認(rèn)你是知道Mysql的安裝方法相速,或者你已經(jīng)有了Mysql數(shù)據(jù)庫碟渺,對(duì)Mysql的安裝配置就不在講了,如果這塊不清楚突诬,請(qǐng)去百度苫拍。
從mysql 站點(diǎn)下載對(duì)應(yīng)的驅(qū)動(dòng)
https://dev.mysql.com/downloads/connector/odbc/
我下載的是8.0.26、8.0.11旺隙、5.3.13
mysql-connector-odbc-8.0.26-linux-glibc2.12-x86-64bit.tar.gz
下載后解壓绒极,我這里為了測試方便,直接將解壓后的目錄重命名使用了蔬捷,沒有將這個(gè)目錄下的bin和lib拷貝到/usr/local/目錄下垄提,具體的操作命令:
tar zxvf mysql-connector-odbc-8.0.26-linux-glibc2.12-x86-64bit.tar.gz
mv mysql-connector-odbc-8.0.26-linux-glibc2.12-x86-64bit mysql-odbc-8.0.26
注冊(cè)Mysql驅(qū)動(dòng)
myodbc-installer -a -d -n "MySQL ODBC 8.0.26 Unicode Driver" -t "Driver=/root/mysql-odbc-8.0.26/lib/libmyodbc8w.so"
myodbc-installer -a -d -n "MySQL ODBC 8.0.26 ANSI Driver" -t "Driver=/root/mysql-odbc-8.0.26/lib/libmyodbc8a.so"
然后查看是否注冊(cè)成功
使用這個(gè)命令:myodbc-installer -d -l
MySQL ODBC 5.0 Unicode Driver
MySQL ODBC 5.0 ANSI Driver
MySQL ODBC 8.0 Unicode Driver
MySQL ODBC 8.0
MySQL ODBC 8.0.26 Unicode Driver
MySQL ODBC 8.0.26 ANSI Driver
這里我安裝了上面說的三個(gè)版本的驅(qū)動(dòng),另外兩個(gè)版本的驅(qū)動(dòng)安裝方式一樣
4.驗(yàn)證通過ODBC訪問Mysql
我們?nèi)ヅ渲肙DBC訪問Mysql的參數(shù)
編輯/etc/odbc.ini文件周拐,加入下面的內(nèi)容铡俐,將信息替換成你自己的
[mysql]
Description = Data source MySQL
Driver = MySQL ODBC 8.0 Unicode Driver
Server = localhost
Host = localhost
Database = demo
Port = 3306
User = root
Password = zhangfeng
然后我們通過:
isql -v mysql
isql -v mysql
+---------------------------------------+
| Connected! |
| |
| sql-statement |
| help [tablename] |
| quit |
| |
+---------------------------------------+
說明我們ODBC配置成功
5.測試Apache Doris ODBC外表
Doris的安裝配置參考我的博客:Apache Doris安裝部署,或者官網(wǎng)的文檔
首先我們?cè)贛ysql數(shù)據(jù)庫見了一個(gè)demo庫及相應(yīng)的表
CREATE TABLE `test_cdc` (
`id` int NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=91234 DEFAULT CHARSET=utf8mb4;
對(duì)應(yīng)的測試數(shù)據(jù)
INSERT INTO `test_cdc` VALUES (123, 'this is a update');
INSERT INTO `test_cdc` VALUES (1212, '測試flink CDC');
INSERT INTO `test_cdc` VALUES (1234, '這是測試');
INSERT INTO `test_cdc` VALUES (11233, 'zhangfeng_1');
INSERT INTO `test_cdc` VALUES (21233, 'zhangfeng_2');
INSERT INTO `test_cdc` VALUES (31233, 'zhangfeng_3');
INSERT INTO `test_cdc` VALUES (41233, 'zhangfeng_4');
INSERT INTO `test_cdc` VALUES (51233, 'zhangfeng_5');
INSERT INTO `test_cdc` VALUES (61233, 'zhangfeng_6');
INSERT INTO `test_cdc` VALUES (71233, 'zhangfeng_7');
INSERT INTO `test_cdc` VALUES (81233, 'zhangfeng_8');
INSERT INTO `test_cdc` VALUES (91233, 'zhangfeng_9');
下面建Doris的ODBC外表妥粟,這里我們是通過ODBC_Resource來創(chuàng)建ODBC外表审丘,這也是推薦的方式,這樣ODBC resource是可以復(fù)用
首先我們?cè)贐E節(jié)點(diǎn)的conf/odbcinst.ini勾给,添加我們的剛才注冊(cè)的8.0.26的ODBC驅(qū)動(dòng)([MySQL ODBC 8.0.26]這部分)
# Driver from the postgresql-odbc package
# Setup from the unixODBC package
[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/psqlodbc.so
Setup = /usr/lib/libodbcpsqlS.so
FileUsage = 1
# Driver from the mysql-connector-odbc package
# Setup from the unixODBC package
[MySQL ODBC 8.0 Unicode Driver]
Description = ODBC for MySQL
Driver = /usr/lib64/libmyodbc8w.so
FileUsage = 1
[MySQL ODBC 5.0 Unicode Driver]
Description = ODBC for MySQL
Driver=/root/mysql-odbc/lib/libmyodbc5w.so
FileUsage = 1
[MySQL ODBC 8.0]
Description = ODBC for MySQL
Driver=/root/mysql-odbc-8/lib/libmyodbc8w.so
FileUsage = 1
[MySQL ODBC 8.0.26]
Description = ODBC for MySQL
Driver=/root/mysql-odbc-8.0.26//lib/libmyodbc8w.so
FileUsage = 1
# Driver from the oracle-connector-odbc package
# Setup from the unixODBC package
[Oracle 19 ODBC driver]
Description=Oracle ODBC driver for Oracle 19
Driver=/usr/lib/libsqora.so.19.1
首先我們創(chuàng)建resource
CREATE EXTERNAL RESOURCE `mysql_8_0_26`
PROPERTIES (
"host" = "localhost",
"port" = "3306",
"user" = "root",
"password" = "zhangfeng",
"database" = "demo",
"table" = "test_cdc",
"driver" = "MySQL ODBC 8.0.26", --注意這里的名稱要和我們上面紅框標(biāo)識(shí)部分的[]里的名稱一致
"odbc_type" = "mysql",
"type" = "odbc_catalog"
);
基于這個(gè)resource創(chuàng)建ODBC外表
CREATE EXTERNAL TABLE `test_odbc_8_0_26` (
`id` int NOT NULL ,
`name` varchar(255) null
) ENGINE=ODBC
COMMENT "ODBC"
PROPERTIES (
"odbc_catalog_resource" = "mysql_8_0_26", --這里的名稱就是我們上面定義的resource的名稱
"database" = "demo",
"table" = "test_cdc"
);
執(zhí)行結(jié)果
mysql> use demo;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> drop table test_odbc_8_0_26;
Query OK, 0 rows affected (0.01 sec)
mysql> CREATE EXTERNAL RESOURCE `mysql_8_0_26`
-> PROPERTIES (
-> "host" = "localhost",
-> "port" = "3306",
-> "user" = "root",
-> "password" = "zhangfeng",
-> "database" = "demo",
-> "table" = "test_cdc",
-> "driver" = "MySQL ODBC 8.0.26",
-> "odbc_type" = "mysql",
-> "type" = "odbc_catalog"
-> );
Query OK, 0 rows affected (0.01 sec)
mysql> CREATE EXTERNAL TABLE `test_odbc_8_0_26` (
-> `id` int NOT NULL ,
-> `name` varchar(255) null
-> ) ENGINE=ODBC
-> COMMENT "ODBC"
-> PROPERTIES (
-> "odbc_catalog_resource" = "mysql_8_0_26",
-> "database" = "demo",
-> "table" = "test_cdc"
-> );
Query OK, 0 rows affected (0.01 sec)
一切顯示正常滩报,下面是見證奇跡的時(shí)候,我們執(zhí)行查詢外表操作
select * from test_odbc_8_0_26;
ERROR 1064 (HY000): errCode = 2, detailMessage = there is no scanNode Backend. [10002: in black list(io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason)]
這個(gè)時(shí)候顯示BE節(jié)點(diǎn)掛了锦秒,我們通過show backends命令去查看也是顯示BE節(jié)點(diǎn)掛了
mysql> show backends\G;
*************************** 1. row ***************************
BackendId: 10002
Cluster: default_cluster
IP: 172.16.192.81
HeartbeatPort: 9050
BePort: 9060
HttpPort: 8040
BrpcPort: 8060
LastStartTime: 2021-12-31 10:43:16
LastHeartbeat: 2021-12-31 11:03:00
Alive: false
SystemDecommissioned: false
ClusterDecommissioned: false
TabletNum: 0
DataUsedCapacity: 0.000
AvailCapacity: 83.224 GB
TotalCapacity: 98.305 GB
UsedPct: 15.34 %
MaxDiskUsedPct: 15.34 %
Tag: {"location" : "default"}
ErrMsg:
Version: 0.15.1-rc09-Unknown
Status: {"lastSuccessReportTabletsTime":"2021-12-31 11:02:22","lastStreamLoadTime":-1}
1 row in set (0.00 sec)
查看BE的日志
顯示是因?yàn)镺DBC問題導(dǎo)致BE宕機(jī)露泊,因?yàn)橹拔以赾entos是沒問題的(但是版本不是這個(gè)是8.0.11)喉镰,這里考慮可能是版本不匹配的問題旅择。后來詢問了社區(qū)這塊的開發(fā)者,得到的答案也確實(shí)是這個(gè)問題侣姆,那么我們換一個(gè)版本試試生真,這個(gè)時(shí)候我首先想到的我不在使用8.x版本的驅(qū)動(dòng)沉噩,使用5.x試試,關(guān)于5.x的驅(qū)動(dòng)安裝柱蟀,我就不在詳細(xì)講解了川蒙,和8.x的安裝方法一樣,參考上面的部分
*** Aborted at 1640918068 (unix time) try "date -d @1640918068" if you are using GNU date ***
PC: @ 0x7f8caaf29b7e (unknown)
*** SIGSEGV (@0x0) received by PID 56420 (TID 0x7f8c62370700) from PID 0; stack trace: ***
@ 0x3022682 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f8cab0f93c0 (unknown)
@ 0x7f8caaf29b7e (unknown)
@ 0x34ef4ac getrn
@ 0x34ef722 lh_insert
@ 0x34abd14 OBJ_NAME_add
@ 0x7f8c09f9d115 ossl_init_ssl_base_ossl_
@ 0x7f8cab0f647f __pthread_once_slow
@ 0x7f8c0a3af194 CRYPTO_THREAD_run_once
@ 0x7f8c09f9cf87 OPENSSL_init_ssl
@ 0x7f8c0a77e504 ssl_start()
@ 0x7f8c0a751f55 mysql_server_init
@ 0x7f8c0a75c425 mysql_init
@ 0x7f8c0a72cbcd DBC::connect()
@ 0x7f8c0a72f403 MySQLDriverConnect()
@ 0x7f8c0a74ea08 SQLDriverConnectW
@ 0x3b060ec SQLDriverConnect
@ 0x1ec046b doris::ODBCConnector::open()
@ 0x1eb9706 doris::OdbcScanNode::open()
@ 0x189e6e9 doris::PlanFragmentExecutor::open_internal()
@ 0x189fb4c doris::PlanFragmentExecutor::open()
@ 0x181b70e doris::FragmentExecState::execute()
@ 0x181f706 doris::FragmentMgr::_exec_actual()
@ 0x1828d4f std::_Function_handler<>::_M_invoke()
@ 0x198a963 doris::ThreadPool::dispatch_thread()
@ 0x1984aac doris::Thread::supervise_thread()
@ 0x7f8cab0ed609 start_thread
@ 0x7f8caaec5293 clone
@ 0x0 (unknown)
下面我們開始測試5.3.13版本的驅(qū)動(dòng)對(duì)Apache Doris外表是否正常长已,我們首先啟動(dòng)這個(gè)BE節(jié)點(diǎn)
關(guān)于5.3.13的ODBC驅(qū)動(dòng)安裝畜眨、注冊(cè)及Doris BE節(jié)點(diǎn)的配置請(qǐng)參考上面的部分,我們直接進(jìn)入測試環(huán)節(jié)
從下圖的測試來看一切正常
mysql> use demo;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> CREATE EXTERNAL RESOURCE `mysql_5`
-> PROPERTIES (
-> "host" = "localhost",
-> "port" = "3306",
-> "user" = "root",
-> "password" = "zhangfeng",
-> "database" = "demo",
-> "table" = "test_cdc",
-> "driver" = "MySQL ODBC 5.0 Unicode Driver",
-> "odbc_type" = "mysql",
-> "type" = "odbc_catalog"
-> );
Query OK, 0 rows affected (0.01 sec)
mysql>
mysql>
mysql> CREATE EXTERNAL TABLE `test_odbc_5` (
-> `id` int NOT NULL ,
-> `name` varchar(255) null
-> ) ENGINE=ODBC
-> COMMENT "ODBC"
-> PROPERTIES (
-> "odbc_catalog_resource" = "mysql_5",
-> "database" = "demo",
-> "table" = "test_cdc"
-> );
Query OK, 0 rows affected (0.01 sec)
mysql> select * from `test_odbc_5`;
+-------+------------------+
| id | name |
+-------+------------------+
| 123 | this is a update |
| 1212 | 測試flink CDC |
| 1234 | 這是測試 |
| 11233 | zhangfeng_1 |
| 21233 | zhangfeng_2 |
| 31233 | zhangfeng_3 |
| 41233 | zhangfeng_4 |
| 51233 | zhangfeng_5 |
| 61233 | zhangfeng_6 |
| 71233 | zhangfeng_7 |
| 81233 | zhangfeng_8 |
| 91233 | zhangfeng_9 |
+-------+------------------+
12 rows in set (0.01 sec)
這個(gè)時(shí)候我在想术瓮,我在centos下8.0.11都是正常的康聂,在ubuntu下這個(gè)版本是不是也是正常的呢,所以我又去測試了8.0.11這個(gè)版本胞四,從測試來看也是正常的恬汁,其他的我就沒再去做測試,可能是8.0.26這個(gè)版本太新了辜伟,還有就是Linux的各種都需要手動(dòng)配置氓侧,這就需要你對(duì)這些非常熟悉,針對(duì)Doris這個(gè)ODBC外表問題出現(xiàn)異常导狡,基本都是你使用的驅(qū)動(dòng)版本不對(duì)约巷,換個(gè)版本在重試
最后針對(duì)Ubuntu下的Apache Doris Mysql ODBC外表的使用Mysql ODBC驅(qū)動(dòng)的版本,推薦:5.3.13和8.0.11這兩個(gè)版本烘豌,你也可以自己嘗試其他版本