author：sufei
說明：組內(nèi)成員發(fā)現(xiàn)的opengauss相關(guān)bug記錄并提交官方
openguass版本 2.0.0
其他版本也發(fā)現(xiàn)該問題
現(xiàn)象說明
主從配置
主庫(kù)
postgres.conf
replconninfo1 = 'localhost=gauss-1-0.gauss-1-headless.gauss-test localport=5433 localservice=5434 remotehost=gauss-1-1.gauss-1-headless.gauss-test remoteport=5433 remoteservice=5434 '
listen_addresses = '*'
remote_read_mode = non_authentication
從庫(kù)
postgres.conf
replconninfo1 = 'localhost=gauss-1-1.gauss-1-headless.gauss-test localport=5433 localservice=5434 remotehost=gauss-1-0.gauss-1-headless.gauss-test remoteport=5433 remoteservice=5434 '
listen_addresses = '*'
remote_read_mode = non_authentication
hba配置,主從都一樣
host all all 0.0.0.0/0 sha256
問題描述
最近在實(shí)現(xiàn)openGuass容器化集群部署時(shí)酝碳，使用statefulset部署openGauss高可用主從集群，由于 pod的IP不是固定的，需要使用域名搭建主從叫编。
在使用域名搭建主從時(shí)，發(fā)現(xiàn)從庫(kù)無法同步主庫(kù)信息砌创，主要表現(xiàn)為
1蚕钦，當(dāng)創(chuàng)建新實(shí)例時(shí)，從庫(kù)啟動(dòng)后連接主庫(kù)進(jìn)行全量備份時(shí)逻恐，提示無法連接數(shù)據(jù)庫(kù)
[teledb@gs-1 data]$ gs_ctl build -D $PGDATA -b full
[2022-03-21 16:35:27.745][340][][gs_ctl]: gs_ctl full build ,datadir is /home/teledb/gsdata/gsroot/data
[2022-03-21 16:35:27.746][340][][gs_ctl]: stop failed, killing gaussdb by force ...
[2022-03-21 16:35:27.746][340][][gs_ctl]: command [ps c -eo pid,euid,cmd | grep gaussdb | grep -v grep | awk '{if($2 == curuid && $1!="-n") print "/proc/"$1"/cwd"}' curuid=`id -u`| xargs ls -l | awk '{if ($NF=="/home/teledb/gsdata/gsroot/data") print $(NF-2)}' | awk -F/ '{print $3 }' | xargs kill -9 >/dev/null 2>&1 ] path: [/home/teledb/gsdata/gsroot/data]
[2022-03-21 16:35:47.872][340][datanode1][gs_ctl]: could not connect to server.
2，使用IP搭建主從后修改為域名峻黍，從庫(kù)以standy啟動(dòng)后复隆，數(shù)據(jù)庫(kù)狀態(tài)永遠(yuǎn)為 Need repair
[2022-03-21 16:29:25.454][353][][gs_ctl]: gs_ctl query
datadir is /home/teledb/gsdata/gsroot/data
HA state:
local_role : Standy
static_connections : 0
db_state : Need repair
查看從庫(kù)啟動(dòng)日志,數(shù)據(jù)目錄下 pg_log目錄，發(fā)現(xiàn)報(bào)錯(cuò)no pg_hba.conf entry for host 姆涩，
排查步驟
- 首先從庫(kù)啟動(dòng)日志, 在數(shù)據(jù)目錄下 pg_log目錄挽拂，發(fā)現(xiàn)報(bào)錯(cuò)no pg_hba.conf entry for host ，但是找不出任何問題
- 然后確認(rèn)使用IP搭建主從是否可行骨饿，發(fā)現(xiàn)使用IP能正常搭建主從亏栈，
- 然后確認(rèn)是不是容器化環(huán)境問題，在物理機(jī)器上配置域名宏赘，搭建主從绒北，并且驗(yàn)證PG試用域名搭建主從， 確認(rèn)域名配置成功置鼻，只是gauss無法使用域名搭建主從镇饮。
- 最后根據(jù)日志分析源碼，定位BUG
根因分析
根據(jù)日志定位到函數(shù)hba_getauthmethod
void hba_getauthmethod(hbaPort* port)
{
/*
* We may receive replication request from gs_basebackup, in this case, use check_hba
* to verify. But if we run gs_basebackup at the machine in ReplConnInfo, we can't
* distinguish whether this request is from a standby process or gs_basebackup...
* In this case, we still need to use check_hba_replication for compatibility.
*/
#ifdef ENABLE_MULTIPLE_NODES
if (IsDSorHaWalSender() ) {
#else
if (IsDSorHaWalSender() && is_node_internal_connection(port)) {
#endif
check_hba_replication(port);
} else {
check_hba(port);
}
}
報(bào)錯(cuò)日志為函數(shù) check_hba調(diào)用報(bào)錯(cuò)，但是連接請(qǐng)求 作為復(fù)制連接储藐，應(yīng)該走check_hba_replication 函數(shù)邏輯
所以這里的判斷函數(shù) IsDSorHaWalSender() && is_node_internal_connection(port) 應(yīng)該返回ture俱济，下面看下這兩個(gè)函數(shù)
bool IsDSorHaWalSender()
{
return (dummyStandbyMode == true || (AM_WAL_SENDER && AM_WAL_DB_SENDER == false));
}
/*
* Check if the current connection is a node internal connection
*/
bool is_node_internal_connection(hbaPort* port)
{
const SockAddr remote_addr = port->raddr;
const SockAddr local_addr = port->laddr;
char remote_host[NI_MAXHOST] = {0};
char local_host[NI_MAXHOST] = {0};
remote_host[0] = '\0';
local_host[0] = '\0';
(void)pg_getnameinfo_all(&remote_addr.addr,
remote_addr.salen,
remote_host,
sizeof(remote_host),
NULL,
0,
NI_NUMERICHOST | NI_NUMERICSERV);
(void)pg_getnameinfo_all(
&local_addr.addr, local_addr.salen, local_host, sizeof(local_host), NULL, 0, NI_NUMERICHOST | NI_NUMERICSERV);
if (strlen(remote_host) == 0) {
return false;
}
if (strcmp(remote_host, local_host) == 0) {
ereport(DEBUG2, (errmsg("remote host is:%s and local host is:%s", remote_host, local_host)));
return true;
}
for (int i = 1; i < MAX_REPLNODE_NUM; i++) {
if (u_sess->attr.attr_storage.ReplConnInfoArr[i] && *(u_sess->attr.attr_storage.ReplConnInfoArr[i]) != '\0' &&
strcasestr(u_sess->attr.attr_storage.ReplConnInfoArr[i], remote_host) != NULL) {
ereport(DEBUG2, (errmsg("remote host is:%s in replconninfo %s",
remote_host, u_sess->attr.attr_storage.ReplConnInfoArr[1])));
return true;
}
}
return false;
}
debug加入上述函數(shù)
Breakpoint 1, is_node_internal_connection (port=0x7fb9f0ad85b0) at hba.cpp:2010
2010 hba.cpp: No such file or directory.
(gdb) p *port
$1 = {sock = 118, noblock = false, proto = 196659, laddr = {addr = {ss_family = 2, __ss_padding = "\025\071\n\364\000\241", '\000' <repeats 111 times>, __ss_align = 0},
salen = 16}, raddr = {addr = {ss_family = 2, __ss_padding = "\353(\n\364\000\242", '\000' <repeats 111 times>, __ss_align = 0}, salen = 16},
remote_host = 0x7fb9ef7e6120 "gs-1.gs-headless.gauss-test.svc.cluster.local", remote_hostname = 0x7fb9ef7e6120 "gs-1.gs-headless.gauss-test.svc.cluster.local",
remote_hostname_resolv = 0, remote_port = 0x7fb9ef7e6188 "60200", libcomm_addrinfo = 0x0, gs_sock = {idx = 0, sid = 0, ver = 0, type = 0},
canAcceptConnections = CAC_OK, database_name = 0x7fb9ef7e6218 "", user_name = 0x7fb9ef7e61d0 "teledb", cmdline_options = 0x7fb9ef7e6260 "-c remotetype=application",
guc_options = 0x7fb9ef7e6378, hba = 0x0, md5Salt = "\235\v\032$", SessionStartTime = 700193972013293, SessionVersionNum = 92298, default_keepalives_idle = 7200,
default_keepalives_interval = 75, default_keepalives_count = 9, keepalives_idle = 7200, keepalives_interval = 75, keepalives_count = 9, gss = 0x7fb9ef67ff98, ssl = 0x0,
peer = 0x0, peer_cn = 0x0, count = 0, token = "\000\000\000\000\000\000\000\000", is_logic_conn = false, msgLog = 0x0, krbsrvname = 0x0, gss_ctx = 0x0, gss_cred = 0x0,
gss_name = 0x0, gss_outbuf = {length = 0, value = 0x0}}
(gdb) p remote_host
$11 = "10.244.0.162", '\000' <repeats 1012 times>
可以發(fā)現(xiàn)remote_host 返回的是IP ，
域名在port->remote_host字段钙勃， 并且函數(shù)沒有進(jìn)行域名校驗(yàn)
所以函數(shù)is_node_internal_connection走到 strcasestr(u_sess->attr.attr_storage.ReplConnInfoArr[i], remote_host) != NULL) 時(shí)蛛碌，無法返回true，導(dǎo)致主從搭建失敗辖源。
修復(fù)
在函數(shù)is_node_internal_connection走到 strcasestr(u_sess->attr.attr_storage.ReplConnInfoArr[i], remote_host) != NULL) 時(shí) 添加域名驗(yàn)證蔚携，比較域名是否在字符串ReplConnInfoArr中
修復(fù)還需考慮域名長(zhǎng)度問題，所以只比較第一個(gè)域名字段克饶，比如gauss-1.scv.local, 只比較第一個(gè)gauss-1酝蜒、修復(fù)函數(shù)如下
/*
* Check if the current connection is a node internal connection
*/
bool is_node_internal_connection(hbaPort* port)
{
const SockAddr remote_addr = port->raddr;
const SockAddr local_addr = port->laddr;
char remote_host[NI_MAXHOST] = {0};
char local_host[NI_MAXHOST] = {0};
char remote_host_prefix[NI_MAXHOST] = {0};
const char *delim = ".";
remote_host[0] = '\0';
local_host[0] = '\0';
(void)pg_getnameinfo_all(&remote_addr.addr,
remote_addr.salen,
remote_host,
sizeof(remote_host),
NULL,
0,
NI_NUMERICHOST | NI_NUMERICSERV);
(void)pg_getnameinfo_all(
&local_addr.addr, local_addr.salen, local_host, sizeof(local_host), NULL, 0, NI_NUMERICHOST | NI_NUMERICSERV);
if (strlen(remote_host) == 0) {
return false;
}
if (strcmp(remote_host, local_host) == 0) {
ereport(DEBUG2, (errmsg("remote host is:%s and local host is:%s", remote_host, local_host)));
return true;
}
strcpy(remote_host_prefix,port->remote_host);
for (int i = 1; i < MAX_REPLNODE_NUM; i++) {
if (u_sess->attr.attr_storage.ReplConnInfoArr[i] && *(u_sess->attr.attr_storage.ReplConnInfoArr[i]) != '\0' &&
(strcasestr(u_sess->attr.attr_storage.ReplConnInfoArr[i], remote_host) != NULL ||
strcasestr(u_sess->attr.attr_storage.ReplConnInfoArr[i], strtok(remote_host_prefix, delim)) != NULL)) {
ereport(DEBUG2, (errmsg("remote host is:%s in replconninfo %s",
remote_host, u_sess->attr.attr_storage.ReplConnInfoArr[1])));
return true;
}
}
return false;
}
提交到社區(qū)
提交到opengauss最新master分支
提交記錄如下：https://gitee.com/opengauss/openGauss-server/pulls/1558
