一 前言

Bottled Water是Confluent公司開發(fā)的一款可以將postgresql數(shù)據(jù)庫轉(zhuǎn)換為kafka events的工具拧粪。主要用于監(jiān)視PG的數(shù)據(jù)增刪改的變化，將變化的數(shù)據(jù)實(shí)時(shí)推送到kafka消息隊(duì)列沧侥，其他關(guān)心數(shù)據(jù)變化的平臺(tái)只要監(jiān)聽kafka的消息可霎，就能得到改變的數(shù)據(jù)。具體參考德哥的博客《實(shí)時(shí)數(shù)據(jù)交換平臺(tái) - BottledWater-pg with confluent》宴杀，本文具體的安裝步驟也參考了《Bottled Water: 實(shí)時(shí)集成postgresql與kafka》癣朗。

二 安裝準(zhǔn)備

安裝完成PostgreSQL，參考《Centos7安裝PostgreSQL9.6》旺罢；
安裝完成Kafka旷余，參考《Kafka部署》；
此外還依賴avro-c扁达，Jansson正卧，libcurl，librdkafka

三 安裝步驟bottledwater-pg

3.1 安裝libjansson

wget http://www.digip.org/jansson/releases/jansson-2.9.tar.bz2
tar -jxvf jansson-2.9.tar.bz2
cd jansson-2.9
./configure --prefix=/home/free/jansson
make
make install

export PKG_CONFIG_PATH=/home/free/jansson/lib/pkgconfig:$PKG_CONFIG_PATH

pkg-config --cflags --libs jansson
-I/home/free/jansson/include  -L/home/free/jansson/lib -ljansson

3.1 安裝avro

wget http://mirrors.hust.edu.cn/apache/avro/avro-1.8.1/avro-src-1.8.1.tar.gz

tar -zxvf avro-src-1.8.1.tar.gz

cd avro-src-1.8.1/lang/c
mkdir build
cd build

cmake .. -DCMAKE_INSTALL_PREFIX=/home/free/avro -DCMAKE_BUILD_TYPE=Release -DTHREADSAFE=true
make
make test
make install
export PKG_CONFIG_PATH=/home/free/avro/lib/pkgconfig:$PKG_CONFIG_PATH

在make這一步驟報(bào)錯(cuò)如下：

opt/avro-src-1.8.1/lang/c/src/schema.c:31:21: fatal error: jansson.h: No such file or directory
 #include "jansson.h"
                     ^
compilation terminated.
make[2]: *** [src/CMakeFiles/avro-shared.dir/schema.o] Error 1
make[1]: *** [src/CMakeFiles/avro-shared.dir/all] Error 2
make: *** [all] Error 2

這一步驟應(yīng)該是編譯缺少了頭文件跪解，先加上去炉旷，然后重新編譯。

export C_INCLUDE_PATH=/home/free/jansson/include:$C_INCLUDE_PATH

3.3 安裝libcurl

wget https://curl.haxx.se/download/curl-7.54.0.tar.gz
tar -zxvf curl-7.54.0.tar.gz
cd curl-7.54.0
 ./configure --prefix=/home/free/curl
make 
make install
export PKG_CONFIG_PATH=/home/free/curl/lib/pkgconfig:$PKG_CONFIG_PATH

3.4 安裝librdkafka

git clone https://github.com/edenhill/librdkafka
cd librdkafka
./configure --prefix=/home/free/librdkafka
#假如報(bào)錯(cuò)如下：
#[root@bogon librdkafka]# ./configure --prefix=/home/freegis/librdkafka
#-bash: ./configure: Permission denied
#賦予執(zhí)行權(quán)限
#[root@bogon librdkafka]# chmod 777 configure

make -j 32
make install

export PKG_CONFIG_PATH=/home/free/librdkafka/lib/pkgconfig:$PKG_CONFIG_PATH

在make的時(shí)候，64位的linux會(huì)報(bào)錯(cuò)如下：

/bin/ld:librdkafka.lds:1: syntax errer in VERSION script

只需要在Makefile.config里面的WITH_LDS=y這一行注釋掉重新make就可以了窘行。

3.5 安裝bottledwater-pg

在環(huán)境變量中新增如下依賴：

vi /etc/profile
#編輯內(nèi)容如下
export CURLHOME=/home/freegis/curl
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$GCCHOME/bin:$CURLHOME/bin
# wq!保存
source /etc/profile

下載安裝文件并編譯部署：

git clone https://github.com/confluentinc/bottledwater-pg

cd bottledwater-pg
#啟用postgres的環(huán)境變量
source /home/postgres/.bashrc
make
make install

make可能報(bào)錯(cuò)如下：

ake[1]: Entering directory `/opt/bottledwater-pg-master/kafka'
gcc bottledwater.c -c -std=c99 -D_POSIX_C_SOURCE=200809L -I../client -I../ext -I/home/postgres/include -I/home/postgres/include/server  -I/home/freegis/avro/include   -DAVRO_1_8    -Wall -Wmissing-prototypes -Wpointer-arith -Wendif-labels -Wmissing-format-attribute -Wformat-security -o bottledwater.o
In file included from table_mapper.h:4:0,
                 from json.h:5,
                 from bottledwater.c:2:
registry.h:4:32: fatal error: librdkafka/rdkafka.h: No such file or directory
 #include <librdkafka/rdkafka.h>
                                ^
compilation terminated.

這種情況仍然是頭文件找不到骏啰，為了麻煩，把用到的頭文件都指定抽高。

export C_INCLUDE_PATH=/home/free/jansson/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/avro/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/librdkafka/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/curl/include:$C_INCLUDE_PATH

繼續(xù)編譯后報(bào)錯(cuò)：

/usr/bin/ld: cannot find -lrdkafka

詳細(xì)如下：

缺失依賴.png

建立依賴軟連接：

ln -s /home/free/librdkafka/lib/librdkafka.so /home/postgres/lib/librdkafka.so

檢查安裝情況

su - postgres
psql
create extension bottledwater;

如果創(chuàng)建擴(kuò)展報(bào)錯(cuò)如下：

mcsas=# create extension bottledwater;
ERROR:  could not load library "/home/postgres/lib/bottledwater.so": libavro.so.23.0.0: cannot open shared object file: No such file or directory

查看下bottledwater.so的依賴：

[postgres@bogon lib]$ ldd bottledwater.so 
    linux-vdso.so.1 =>  (0x00007fffc41fe000)
    libavro.so.23.0.0 => /home/freegis/avro/lib/libavro.so.23.0.0 (0x00007f1f26769000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f1f263a8000)
    libjansson.so.4 => /lib64/libjansson.so.4 (0x00007f1f2619c000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1f25f7f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1f26bba000)

發(fā)現(xiàn)所有的依賴都正確。
后經(jīng)大神提示透绩，重啟數(shù)據(jù)庫翘骂，然后執(zhí)行擴(kuò)展。

pg_ctl restart
psql
\c mcsas
create extension bottledwater;
CREATE EXTENSION