author：sufei
版本：mysql 8.0.18蜈项，mysql 8.0.20
現(xiàn)象：
Mysql 啟動(dòng)時(shí)出現(xiàn)coredump
錯(cuò)誤日志如下：
2020-07-01T14:46:04.895732+08:00 0 [Note] [MY-010251] [Server] Server socket created on IP: '::'.
2020-07-01T14:46:04.895732+08:00 0 [Note] [MY-010251] [Server] Server socket created on IP: '::'.
06:46:04 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x6224e90
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 2ba1bff42c80 thread_stack 0x40000
/mysql-install/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x1e825be]
/mysql-install/bin/mysqld(handle_fatal_signal+0x341) [0xf8b031]
/lib64/libpthread.so.0(+0xf630) [0x2ba0ceb75630]
/mysql-install/bin/mysqld(Cached_authentication_plugins::get_cached_plugin_ref(MYSQL_LEX_CSTRING const*)+0xd) [0xfd6e6d]
/mysql-install/bin/mysqld() [0xfd6f2c]
/mysql-install/bin/mysqld(acl_authenticate(THD*, enum_server_command)+0x1f7) [0xfe03f7]
/mysql-install/bin/mysqld() [0xdf1635]
/mysql-install/bin/mysqld(thd_prepare_connection(THD*)+0x46) [0xdf27e6]
/mysql-install/bin/mysqld() [0xf78e41]
/mysql-install/bin/mysqld() [0x23c08a5]
/lib64/libpthread.so.0(+0x7ea5) [0x2ba0ceb6dea5]
/lib64/libc.so.6(clone+0x6d) [0x2ba0d087e8dd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 4
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
Core文件調(diào)用棧如下：
(gdb) bt
#0 0x00002ba0ceb72aa1 in pthread_kill () from /lib64/libpthread.so.0
#1 0x0000000001e81c47 in my_write_core (sig=<optimized out>) at /data2/sf/mysql8/mysql/8.0.18/mysql-8.0.18/mysys/stacktrace.cc:305
#2 0x0000000000f8afdd in handle_fatal_signal (sig=11) at /data2/sf/mysql8/mysql/8.0.18/mysql-8.0.18/sql/signal_handler.cc:169
#3 <signal handler called>
#4 0x0000000000fd6e6d in Cached_authentication_plugins::get_cached_plugin_ref (this=0x0, plugin=plugin@entry=0x2ba1bff41ff0)
at /data2/sf/mysql8/mysql/8.0.18/mysql-8.0.18/sql/auth/sql_authentication.cc:881
#5 0x0000000000fd6f2c in do_auth_once (thd=thd@entry=0x6224e90, auth_plugin_name=..., mpvio=mpvio@entry=0x2ba1bff423c0)
at /data2/sf/mysql8/mysql/8.0.18/mysql-8.0.18/sql/auth/sql_authentication.cc:2971
#6 0x0000000000fe03f7 in acl_authenticate (thd=thd@entry=0x6224e90, command=command@entry=COM_CONNECT)
at /data2/sf/mysql8/mysql/8.0.18/mysql-8.0.18/sql/auth/sql_authentication.cc:3268
#7 0x0000000000df1635 in check_connection (thd=thd@entry=0x6224e90, this=<optimized out>) at /data2/sf/mysql8/mysql/8.0.18/mysql-8.0.18/sql/sql_connect.cc:649
#8 0x0000000000df27e6 in login_connection (thd=0x6224e90) at /data2/sf/mysql8/mysql/8.0.18/mysql-8.0.18/sql/sql_connect.cc:704
#9 thd_prepare_connection (thd=thd@entry=0x6224e90) at /data2/sf/mysql8/mysql/8.0.18/mysql-8.0.18/sql/sql_connect.cc:877
#10 0x0000000000f78e41 in handle_connection (arg=arg@entry=0x2ba0e42e6970) at /data2/sf/mysql8/mysql/8.0.18/mysql-8.0.18/sql/conn_handler/connection_handler_per_thread.cc:298
#11 0x00000000023c08a5 in pfs_spawn_thread (arg=0x2ba0e4752210) at /data2/sf/mysql8/mysql/8.0.18/mysql-8.0.18/storage/perfschema/pfs.cc:2854
#12 0x00002ba0ceb6dea5 in start_thread () from /lib64/libpthread.so.0
#13 0x00002ba0d087e8dd in clone () from /lib64/libc.so.6
從core文件中可以看出：
1伯铣、發(fā)送coredump的位置是在等入時(shí)玉凯，進(jìn)行權(quán)限認(rèn)證构韵；
2尝丐、然而權(quán)限認(rèn)證的全局變量g_cached_authentication_plugins為空指針蔚约，從而造成段錯(cuò)誤
分析：
?通過上面的分析可以知道坑赡，也就是在使能監(jiān)聽用戶連接時(shí)烙如，acl權(quán)限變量還沒有初始化，這在通常是不可能的毅否，因?yàn)椋?/p>
- 全局變量g_cached_authentication_plugins的初始化是在acl_init函數(shù)中進(jìn)行的亚铁，而該函數(shù)的調(diào)用位置為6738行，如下：
if (abort || acl_init(opt_noacl)) {
if (!abort) LogErr(ERROR_LEVEL, ER_PRIVILEGE_SYSTEM_INIT_FAILED);
abort = true;
opt_noacl = true;
}
- 傳統(tǒng)的監(jiān)聽是在主函數(shù)的7050行
mysqld_socket_acceptor->connection_event_loop();
?也就是在監(jiān)聽之前已經(jīng)初始化了g_cached_authentication_plugins
?這不應(yīng)該出問題呀螟加，可是在mysql8.0.18開啟了admin_IP的獨(dú)立線程徘溢，即設(shè)置create_admin_listener_thread =ON時(shí)，我們可以看到其監(jiān)聽的開啟在network_init中（位置為主函數(shù)的6671捆探，在acl_init之前）然爆，其調(diào)用棧如下：
network_init
|->mysqld_socket_acceptor->init_connection_acceptor()
|--->Mysqld_socket_listener->setup_listener
|------>spawn_admin_thread(m_admin_interface_listen_socket,
m_admin_bind_address.network_namespace) // 開啟監(jiān)聽線程
所以在設(shè)置create_admin_listener_thread = ON時(shí)，存在監(jiān)聽線程先于acl_init初始化黍图，此時(shí)（network_init調(diào)用與acl_init調(diào)用之間）如果有admin用戶使用admin_port端口連接進(jìn)來就會(huì)造成cordump曾雕。
復(fù)現(xiàn)：
開啟一個(gè)循環(huán)連接數(shù)據(jù)庫的腳本，模擬客戶端連接（注意用戶名隨便助被，無需登錄）
#!/bin/bash
for a in {1..10000}
do
/mysql-install/bin/mysql -h admin_ip -Padmin_port -uroot -pxxxxx
done
同時(shí)剖张，配置文件中開啟create_admin_listener_thread ，啟動(dòng)數(shù)據(jù)庫恰起。
在不斷的重復(fù)可能還出現(xiàn)如下問題修械，即任意用戶登入成功，跳過了權(quán)限檢測(cè)检盼。
image.png
出現(xiàn)這個(gè)問題的原因主要是由于在acl_init中初始化g_cached_authentication_plugins與initialized之間
g_cached_authentication_plugins = new Cached_authentication_plugins(); // 初始化g_cached_authentication_plugins
unknown_accounts = new Map_with_rw_lock<Auth_id, uint>(0);
if (!g_cached_authentication_plugins->is_valid()) return 1;
if (dont_read_acl_tables) {
return 0; /* purecov: tested */
}
if (!(thd = new THD)) return 1; /* purecov: inspected */
thd->thread_stack = (char *)&thd;
thd->store_globals();
return_val = check_engine_type_for_acl_table(thd, false);
check_acl_tables_intact(thd, false);
return_val |= acl_reload(thd, false); // 設(shè)置initialized為true
如果啟動(dòng)時(shí)主線程在兩個(gè)變量初始化之間箫章，而管理的監(jiān)控線程可以通過權(quán)限認(rèn)證铐料，邏輯如下
// 在acl_authenticate函數(shù)中
do_auth_once(thd, auth_plugin_name, &mpvio); // 由于g_cached_authentication_plugins設(shè)置了拾并，所以可以通過而不發(fā)生coredump
……
if (initialized) // 由于此時(shí)initialized依然為false
{
……
}else{
sctx->skip_grants(); //跳過了權(quán)限表誓酒，從而造成用戶跳過權(quán)限表登入
}
修復(fù)：
之前團(tuán)隊(duì)已經(jīng)向官方提bug，可以通過關(guān)閉create_admin_listener_thread 來規(guī)避貌亭。后續(xù)在mysql 8.0.22進(jìn)行了修復(fù)柬唯。
image.png
