0.存在的疑惑
- glusterd啟動(dòng)后怎么把glusterfsd的brick進(jìn)程拉起來(lái)隘膘？
- glusterfsd啟動(dòng)的參數(shù)從哪里獲取？
- glusterfsd如何加載的呢朱灿？
- glusterfsd和glusterd啟動(dòng)后各自都是什么角色？(后續(xù)再討論)
1. glusterfs的入口函數(shù)解釋
- glusterfsd中main方法加載xlator {name = 0x687dd0 "management", type = 0x687f50 "mgmt/glusterd"}聚谁，然后fork一個(gè)子進(jìn)程加載mgmt/glusterd中的init方法
2.glusterd啟動(dòng)過(guò)程
- 子進(jìn)程中調(diào)用mgmt/glusterd對(duì)應(yīng)源文件中g(shù)lusterd.c中的init,針對(duì)glusterd進(jìn)行初始化母剥，依次會(huì)調(diào)用如下方法:
- glusterfs_volumes_init
- glusterd.c:init
- glusterd_spawn_daemons
- glusterd_restart_bricks
- glusterd_brick_start
- glusterd_volume_start_glusterfs
- runner_run
- runner_start:fork子進(jìn)程，調(diào)用execvp加載/usr/local/sbin/glusterfsd二進(jìn)制文件
int runner_start(runner_t *runner) { //-----忽略---------------- execvp(runner->argv[0], runner->argv); //-----忽略---------------- }
3.glusterfsd如何加載xlator
-
glusterfsd是通過(guò)讀取配置文件/var/lib/glusterd/vols/dht_vol/dht_vol.172.17.0.2.data-brick.vol來(lái)加載glusterfsd運(yùn)行期間需要的xlator，這些配置信息是用戶創(chuàng)建volume時(shí)候系統(tǒng)生成的
截屏2020-02-21下午8.03.09.png
4. glusterd啟動(dòng)后進(jìn)程列表
-
runner_start后繼續(xù)glusterfsd.c中的main方法环疼，啟動(dòng)對(duì)應(yīng)的服務(wù)习霹，glusterfsd正式啟動(dòng)了，就如systemctl start glusterd后出現(xiàn)如下的結(jié)果:
image.png
5.glusterfsd啟動(dòng)流程
- 上個(gè)步驟通過(guò)runner_start加載glusterfsd的二進(jìn)制進(jìn)程炫隶，glusterfsd進(jìn)入初始化節(jié)點(diǎn)淋叶，首先 glusterfs_listener_init 初始化tcp連接
- glusterfsd通過(guò)tcp連接到glusterd獲取volume xlator信息，準(zhǔn)備構(gòu)建xlator的graph.
volume xlator信息如下:
(gdb) p rsp.spec
$6 = 0x7fffe4002fc0 "volume dht_vol-posix\n type storage/posix\n option shared-brick-count 1\n option volume-id 25305c2d-317f-45e9-8e60-be322ed69ee1\n option directory /data/brick\nend-volume\n\nvolume dht_vol-trash\n type features/trash\n option trash-internal-op off\n option brick-path /data/brick\n option trash-dir .trashcan\n subvolumes dht_vol-posix\nend-volume\n\nvolume dht_vol-changelog\n type features/changelog\n option changelog-barrier-timeout 120\n option changelog-dir /data/brick/.glusterfs/changelogs\n option changelog-brick /data/brick\n subvolumes dht_vol-trash\nend-volume\n\nvolume dht_vol-bitrot-stub\n type features/bitrot-stub\n option bitrot disable\n option export /data/brick\n subvolumes dht_vol-changelog\nend-volume\n\nvolume dht_vol-access-control\n type features/access-control\n subvolumes dht_vol-bitrot-stub\nend-volume\n\nvolume dht_vol-locks\n type features/locks\n option enforce-mandatory-lock off\n subvolumes dht_vol-access-control\nend-volume\n\nvolume dht_vol-worm\n type features/worm\n option worm-files-deletable on\n option worm-file-level off\n option worm off\n subvolumes dht_vol-locks\nend-volume\n\nvolume dht_vol-read-only\n type features/read-only\n option read-only off\n subvolumes dht_vol-worm\nend-volume\n\nvolume dht_vol-leases\n type features/leases\n option leases off\n subvolumes dht_vol-read-only\nend-volume\n\nvolume dht_vol-upcall\n type features/upcall\n option cache-invalidation off\n subvolumes dht_vol-leases\nend-volume\n\nvolume dht_vol-io-threads\n type performance/io-threads\n subvolumes dht_vol-upcall\nend-volume\n\nvolume dht_vol-selinux\n type features/selinux\n option selinux on\n subvolumes dht_vol-io-threads\nend-volume\n\nvolume dht_vol-marker\n type features/marker\n option inode-quota off\n option quota off\n option gsync-force-xtime off\n option xtime off\n option quota-version 0\n option timestamp-file /var/lib/glusterd/vols/dht_vol/marker.tstamp\n option volume-uuid 25305c2d-317f-45e9-8e60-be322ed69ee1\n subvolumes dht_vol-selinux\nend-volume\n\nvolume dht_vol-barrier\n type features/barrier\n option barrier-timeout 120\n option barrier disable\n subvolumes dht_vol-marker\nend-volume\n\nvolume dht_vol-index\n type features/index\n option xattrop-pending-watchlist trusted.afr.dht_vol-\n option xattrop-dirty-watchlist trusted.afr.dirty\n option index-base /data/brick/.glusterfs/indices\n subvolumes dht_vol-barrier\nend-volume\n\nvolume dht_vol-quota\n type features/quota\n option deem-statfs off\n option server-quota off\n option volume-uuid dht_vol\n subvolumes dht_vol-index\nend-volume\n\nvolume dht_vol-io-stats\n type debug/io-stats\n option count-fop-hits off\n option latency-measurement off\n option log-level INFO\n option unique-id /data/brick\n subvolumes dht_vol-quota\nend-volume\n\nvolume /data/brick\n type performance/decompounder\n subvolumes dht_vol-io-stats\nend-volume\n\nvolume dht_vol-server\n type protocol/server\n option transport.listen-backlog 1024\n option transport.socket.keepalive-count 9\n option transport.socket.keepalive-interval 2\n option transport.socket.keepalive-time 20\n option transport.socket.ssl-enabled off\n option transport.socket.keepalive 1\n option auth.addr./data/brick.allow *\n option auth-path /data/brick\n option auth.login.b26ac1fd-40a4-4c6e-b211-a9f2b58beedc.password 4eda9afe-9466-4283-b175-6020d5981994\n option auth.login./data/brick.allow b26ac1fd-40a4-4c6e-b211-a9f2b58beedc\n option transport.address-family inet\n option transport-type tcp\n subvolumes /data/brick\nend-volume\n"
- 根據(jù)xlator獲取初構(gòu)建graph信息伪阶，初始化每個(gè)xlator的init方法煞檩，這個(gè)時(shí)候glusterfsd初始化成功
6. gdb 調(diào)試方法
- gdb break 信息
set print pretty on
//調(diào)試子進(jìn)程,子進(jìn)程會(huì)加載執(zhí)行mgmt/glusterd/src/glusterd.c:init
set follow-fork-mode child
set detach-on-fork off
br glusterfsd.c:main
br br glusterfs_volumes_init //glusterd和glusterfsd都會(huì)調(diào)用該方法
br glusterd.c:init
//------ 進(jìn)入init函數(shù)后執(zhí)行如下------
/* 進(jìn)入這個(gè)函數(shù)后設(shè)置調(diào)試父進(jìn)程
set follow-fork-mode parent
set detach-on-fork on
*/
br glusterd_spawn_daemons
br glusterd_restart_bricks
br glusterd_brick_start
br glusterd_volume_start_glusterfs
/* 進(jìn)入這個(gè)函數(shù)后設(shè)置調(diào)試父進(jìn)程
set follow-fork-mode child
set detach-on-fork on
*/
br runner_run
br runner_log
br runner_start
/** 進(jìn)入runner_start 后設(shè)置如下斷點(diǎn)，glusterfsd啟動(dòng)以后會(huì)按照如下的調(diào)用鏈進(jìn)行 */
br glusterfs_volumes_init
br glusterfs_listener_init //glusterd不會(huì)調(diào)用該方法
br glusterfs_mgmt_init //glusterd不會(huì)調(diào)用該方法
br mgmt_rpc_notify
br glusterfs_volfile_fetch
br glusterfs_volfile_fetch_one
br mgmt_getspec_cbk
br glusterfs_process_volfp
br glusterfs_graph_activate
br glusterfs_graph_init
- 1.glusterd gdb日志
[root@b94a78ebfc80 ~]$ gdb glusterd
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/sbin/glusterfsd...done.
(gdb) br glusterfsd.c:main
Breakpoint 1 at 0x40b71d: file glusterfsd.c, line 2746.
(gdb) br glusterd.c:init
No source file named glusterd.c.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (glusterd.c:init) pending.
(gdb) set print pretty on
(gdb) set follow-fork-mode child
(gdb) set detach-on-fork off
(gdb) c
The program is not being run.
(gdb) r
Starting program: /usr/local/sbin/glusterd
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Breakpoint 1, main (argc=1, argv=0x7fffffffe778) at glusterfsd.c:2746
2746 glusterfs_ctx_t *ctx = NULL;
Missing separate debuginfos, use: debuginfo-install glibc-2.17-292.el7.x86_64 libuuid-2.23.2-61.el7_7.1.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) c
Continuing.
[New process 956]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff5431700 (LWP 957)]
[New Thread 0x7ffff4c30700 (LWP 958)]
[New Thread 0x7ffff442f700 (LWP 959)]
[New Thread 0x7ffff3c2e700 (LWP 960)]
[New Thread 0x7ffff342d700 (LWP 961)]
[New Thread 0x7ffff2c2c700 (LWP 962)]
Reading symbols from /usr/local/lib/glusterfs//xlator/mgmt/glusterd.so...done.
[Switching to Thread 0x7ffff7fea4c0 (LWP 956)]
Breakpoint 2, init (this=0x686d40) at glusterd.c:1373
1373 int32_t ret = -1;
Missing separate debuginfos, use: debuginfo-install glibc-2.17-292.el7.x86_64 libuuid-2.23.2-61.el7_7.1.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 userspace-rcu-0.7.16-1.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) set follow-fork-mode parent
(gdb) set detach-on-fork on
(gdb) br glusterd_spawn_daemons
Breakpoint 3 at 0x7ffff1f79f85: file glusterd-utils.c, line 3619.
(gdb) br glusterd_restart_bricks
Breakpoint 4 at 0x7ffff1f832e3: file glusterd-utils.c, line 6334.
(gdb) br glusterd_brick_start
Breakpoint 5 at 0x7ffff1f826cd: file glusterd-utils.c, line 6107.
(gdb) br glusterd_volume_start_glusterfs
Breakpoint 6 at 0x7ffff1f73f5f: file glusterd-utils.c, line 2024.
(gdb) c
Continuing.
Reading symbols from /usr/local/lib/glusterfs//rpc-transport/socket.so...done.
Reading symbols from /usr/local/lib/glusterfs//rpc-transport/rdma.so...done.
Detaching after fork from child process 964.
Detaching after fork from child process 965.
Detaching after fork from child process 966.
Detaching after fork from child process 967.
Detaching after fork from child process 968.
Detaching after fork from child process 969.
Detaching after fork from child process 970.
Detaching after fork from child process 971.
Detaching after fork from child process 972.
Detaching after fork from child process 973.
Detaching after fork from child process 974.
Detaching after fork from child process 975.
Detaching after fork from child process 976.
Detaching after fork from child process 977.
Detaching after fork from child process 978.
Detaching after fork from child process 979.
Detaching after fork from child process 980.
Detaching after fork from child process 981.
Detaching after fork from child process 982.
Detaching after fork from child process 983.
Detaching after fork from child process 984.
Detaching after fork from child process 985.
Detaching after fork from child process 986.
Detaching after fork from child process 987.
Detaching after fork from child process 988.
[Switching to Thread 0x7ffff342d700 (LWP 961)]
Breakpoint 3, glusterd_spawn_daemons (opaque=0x0) at glusterd-utils.c:3619
3619 glusterd_conf_t *conf = THIS->private;
Missing separate debuginfos, use: debuginfo-install keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_7.2.x86_64 libcom_err-1.42.9-16.el7.x86_64 libibverbs-22.1-3.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 librdmacm-22.1-3.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 pcre-8.32-17.el7.x86_64
(gdb) n
3620 int ret = -1;
(gdb)
3622 synclock_lock(&conf->big_lock);
(gdb)
[New Thread 0x7fffef606700 (LWP 989)]
3623 glusterd_restart_bricks();
(gdb)
Breakpoint 4, glusterd_restart_bricks (opaque=0x7fffef9a9fc8) at glusterd-utils.c:6334
6334 int ret = 0;
(gdb)
6335 glusterd_volinfo_t *volinfo = NULL;
(gdb)
6336 glusterd_brickinfo_t *brickinfo = NULL;
(gdb) set follow-fork-mode child
(gdb) set detach-on-fork on
(gdb) br runner_run
Breakpoint 7 at 0x7ffff7b35f04: runner_run. (2 locations)
(gdb) br runner_start
Breakpoint 8 at 0x7ffff7b35825: runner_start. (2 locations)
(gdb) info break
Num Type Disp Enb Address What
1 breakpoint keep y <MULTIPLE>
breakpoint already hit 1 time
1.1 y 0x000000000040b71d in main at glusterfsd.c:2746 inf 1
1.2 y 0x000000000040b71d in main at glusterfsd.c:2746 inf 2
2 breakpoint keep y 0x00007ffff1f3a7ee in init at glusterd.c:1373 inf 2
breakpoint already hit 1 time
3 breakpoint keep y 0x00007ffff1f79f85 in glusterd_spawn_daemons at glusterd-utils.c:3619 inf 2
breakpoint already hit 1 time
4 breakpoint keep y 0x00007ffff1f832e3 in glusterd_restart_bricks at glusterd-utils.c:6334 inf 2
breakpoint already hit 1 time
5 breakpoint keep y 0x00007ffff1f826cd in glusterd_brick_start at glusterd-utils.c:6107 inf 2
6 breakpoint keep y 0x00007ffff1f73f5f in glusterd_volume_start_glusterfs at glusterd-utils.c:2024 inf 2
7 breakpoint keep y <MULTIPLE>
7.1 y 0x00007ffff7b35f04 in runner_run at run.c:430 inf 1
7.2 y 0x00007ffff7b35f04 in runner_run at run.c:430 inf 2
8 breakpoint keep y <MULTIPLE>
8.1 y 0x00007ffff7b35825 in runner_start at run.c:259 inf 1
8.2 y 0x00007ffff7b35825 in runner_start at run.c:259 inf 2
(gdb) c
Continuing.
Breakpoint 5, glusterd_brick_start (volinfo=0x6db4b0, brickinfo=0x6ef490, wait=false, only_connect=false) at glusterd-utils.c:6107
6107 int ret = -1;
(gdb) c
Continuing.
[New Thread 0x7fffeee05700 (LWP 991)]
Breakpoint 6, glusterd_volume_start_glusterfs (volinfo=0x6db4b0, brickinfo=0x6ef490, wait=false) at glusterd-utils.c:2024
2024 int32_t ret = -1;
(gdb) c
Continuing.
[New process 992]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Switching to Thread 0x7ffff342d700 (LWP 992)]
Breakpoint 8, runner_start (runner=0x7fffef800e00) at run.c:259
259 int pi[3][2] = {{-1, -1}, {-1, -1}, {-1, -1}};
(gdb) n
261 int ret = 0;
(gdb)
262 int errno_priv = 0;
(gdb)
263 int i = 0;
(gdb)
266 if (runner->runerr || !runner->argv) {
(gdb)
271 GF_ASSERT(runner->argv[0]);
(gdb) set follow-fork-mode child
(gdb) set detach-on-fork on
(gdb) n
276 ret = pipe(xpi);
(gdb)
277 if (ret != -1)
(gdb)
278 ret = fcntl(xpi[1], F_SETFD, FD_CLOEXEC);
(gdb)
280 for (i = 0; i < 3; i++) {
(gdb)
281 if (runner->chfd[i] != -2)
(gdb)
282 continue;
(gdb)
280 for (i = 0; i < 3; i++) {
(gdb)
281 if (runner->chfd[i] != -2)
(gdb)
282 continue;
(gdb)
280 for (i = 0; i < 3; i++) {
(gdb)
281 if (runner->chfd[i] != -2)
(gdb)
282 continue;
(gdb)
280 for (i = 0; i < 3; i++) {
(gdb)
291 if (ret != -1)
(gdb)
292 runner->chpid = fork();
(gdb)
[New process 995]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Switching to Thread 0x7ffff342d700 (LWP 995)]
293 switch (runner->chpid) {
327 if (ret != -1) {
(gdb)
328 int fdv[4] = {0, 1, 2, xpi[1]};
(gdb)
330 ret = close_fds_except(fdv, sizeof(fdv) / sizeof(*fdv));
(gdb)
333 if (ret != -1) {
(gdb)
335 sigemptyset(&set);
(gdb)
336 sigprocmask(SIG_SETMASK, &set, NULL);
(gdb)
338 execvp(runner->argv[0], runner->argv);
(gdb) p runner->argv[0]
$1 = 0x7fffe8000c60 "/usr/local/sbin/glusterfsd"
(gdb) p runner->argv
$2 = (char **) 0x7fffe8000970
(gdb) p/s runner->argv
$3 = (char **) 0x7fffe8000970
(gdb) p/s runner->argv[1]
$4 = 0x7fffe8000ce0 "-s"
(gdb) p/s runner->argv[2]
$5 = 0x7fffe8000d50 "172.17.0.2"
(gdb) p/s runner->argv[3]
$6 = 0x7fffe8000dc0 "--volfile-id"
(gdb) p/s runner->argv[4]
$7 = 0x7fffe8000e30 "dht_vol.172.17.0.2.data-brick"
(gdb) p/s runner->argv[5]
$8 = 0x7fffe8000eb0 "-p"
(gdb) p/s runner->argv[6]
$9 = 0x7fffe8000f20 "/var/run/gluster/vols/dht_vol/172.17.0.2-data-brick.pid"
(gdb) p/s runner->argv[7]
$10 = 0x7fffe8000fc0 "-S"
(gdb) p/s runner->argv[8]
$11 = 0x7fffe8001030 "/var/run/gluster/cb954c564295c3be.socket"
(gdb) p/s runner->argv[9]
$12 = 0x7fffe80010c0 "--brick-name"
(gdb) p/s runner->argv[10]
$13 = 0x7fffe8001130 "/data/brick"
(gdb) p/s runner->argv[11]
$14 = 0x7fffe80011a0 "-l"
(gdb) p/s runner->argv[12]
$15 = 0x7fffe8001210 "/var/log/glusterfs/bricks/data-brick.log"
(gdb) p/s runner->argv[13]
$16 = 0x7fffe80012a0 "--xlator-option"
(gdb) p/s runner->argv[14]
$17 = 0x7fffe8001310 "*-posix.glusterd-uuid=88efffbe-ce9e-4117-8d1a-77de0a8ecd75"
(gdb) p/s runner->argv[15]
$18 = 0x7fffe80013b0 "--process-name"
(gdb) p/s runner->argv[16]
$19 = 0x7fffe8001420 "brick"
(gdb) p/s runner->argv[17]
$20 = 0x7fffe8001490 "--brick-port"
(gdb) p/s runner->argv[18]
$21 = 0x7fffe8001500 "49152"
(gdb) p/s runner->argv[19]
$22 = 0x7fffe8001570 "--xlator-option"
(gdb) p/s runner->argv[20]
$23 = 0x7fffe80015e0 "dht_vol-server.listen-port=49152"
(gdb) p/s runner->argv[21]
$24 = 0x0
(gdb)
- glusterfsd gdb 日志
(gdb) info break
Num Type Disp Enb Address What
1 breakpoint keep y <MULTIPLE>
breakpoint already hit 2 times
1.1 y 0x000000000040b71d in main at glusterfsd.c:2746 inf 1
1.2 y 0x000000000040b71d in main at glusterfsd.c:2746 inf 5, 4, 3
2 breakpoint keep y <PENDING> glusterd.c:init inf 5, 4, 3
breakpoint already hit 1 time
3 breakpoint keep y <PENDING> glusterd_spawn_daemons inf 5, 4, 3
breakpoint already hit 1 time
4 breakpoint keep y <PENDING> glusterd_restart_bricks inf 5, 4, 3
breakpoint already hit 1 time
5 breakpoint keep y <PENDING> glusterd_brick_start inf 5, 4, 3
breakpoint already hit 1 time
6 breakpoint keep y <PENDING> glusterd_volume_start_glusterfs inf 5, 4, 3
breakpoint already hit 1 time
7 breakpoint keep y <MULTIPLE>
7.1 y 0x00007ffff7b35f04 in runner_run at run.c:430 inf 1
7.2 y 0x00007ffff7b35f04 in runner_run at run.c:430 inf 5, 4, 3
8 breakpoint keep y <MULTIPLE>
breakpoint already hit 1 time
8.1 y 0x00007ffff7b35825 in runner_start at run.c:259 inf 1
8.2 y 0x00007ffff7b35825 in runner_start at run.c:259 inf 5, 4, 3
9 breakpoint keep y <PENDING> server-rpc-fops_v2.c:server4_0_mkdir
10 breakpoint keep y <PENDING> server-resolve.c:server_resolve
11 breakpoint keep y <MULTIPLE>
breakpoint already hit 1 time
11.1 y 0x000000000040b603 in glusterfs_volumes_init at glusterfsd.c:2704 inf 1
11.2 y 0x000000000040b603 in glusterfs_volumes_init at glusterfsd.c:2704 inf 5, 4, 3
12 breakpoint keep y <MULTIPLE>
12.1 y 0x000000000040b390 in glusterfs_process_volfp at glusterfsd.c:2626 inf 1
12.2 y 0x000000000040b390 in glusterfs_process_volfp at glusterfsd.c:2626 inf 5, 4, 3
13 breakpoint keep y <MULTIPLE>
13.1 y 0x000000000040710f in get_volfp at glusterfsd.c:781 inf 1
13.2 y 0x000000000040710f in get_volfp at glusterfsd.c:781 inf 5, 4, 3
14 breakpoint keep y <MULTIPLE>
breakpoint already hit 1 time
14.1 y 0x0000000000412581 in glusterfs_mgmt_init at glusterfsd-mgmt.c:2648 inf 1
14.2 y 0x0000000000412581 in glusterfs_mgmt_init at glusterfsd-mgmt.c:2648 inf 5, 4, 3
15 breakpoint keep y <MULTIPLE>
breakpoint already hit 1 time
---Type <return> to continue, or q <return> to quit---
15.1 y 0x0000000000411848 in glusterfs_volfile_fetch at glusterfsd-mgmt.c:2270 inf 1
15.2 y 0x0000000000411848 in glusterfs_volfile_fetch at glusterfsd-mgmt.c:2270 inf 5, 4, 3
16 breakpoint keep y <MULTIPLE>
breakpoint already hit 1 time
16.1 y 0x00000000004114ed in glusterfs_volfile_fetch_one at glusterfsd-mgmt.c:2166 inf 1
16.2 y 0x00000000004114ed in glusterfs_volfile_fetch_one at glusterfsd-mgmt.c:2166 inf 5, 4, 3
17 breakpoint keep y <MULTIPLE>
breakpoint already hit 1 time
17.1 y 0x0000000000410a85 in mgmt_getspec_cbk at glusterfsd-mgmt.c:1922 inf 1
17.2 y 0x0000000000410a85 in mgmt_getspec_cbk at glusterfsd-mgmt.c:1922 inf 5, 4, 3
18 breakpoint keep y <MULTIPLE>
18.1 y 0x000000000040b390 in glusterfs_process_volfp at glusterfsd.c:2626 inf 1
18.2 y 0x000000000040b390 in glusterfs_process_volfp at glusterfsd.c:2626 inf 5, 4, 3
(gdb)
