某次做checklist赌莺,當看到機器的監(jiān)控時窖逗,發(fā)現下午某時刻某臺機器突發(fā)大量CLOSE_WAIT累奈。一下子就懷疑是服務端處理超時了,client主動斷開連接造成的彻桃。然后定位到某個服務下午上線,由于db大量阻塞導致的問題晾蜘。數據庫不是有超時么邻眷,連接和讀寫都有啊剔交?肆饶?
其實這個問題網上也有很多網友反饋,只不過我們這次切切實實的踩到坑了岖常。go1.8也基本修復了這個問題驯镊。下面來看看到底是怎么回事。我們看database/sql這個包就行了。
先看1.7 怎么獲取連接的:
// Out of free connections or we were asked not to use one. If we're not // allowed to open any more connections, make a request and wait. if db.maxOpen > 0 && db.numOpen >= db.maxOpen { // Make the connRequest channel. It's buffered so that the // connectionOpener doesn't block while waiting for the req to be read. req := make(chan connRequest, 1) db.connRequests = append(db.connRequests, req) db.mu.Unlock() ret, ok := <-req if !ok { return nil, errDBClosed } if ret.err == nil && ret.conn.expired(lifetime) { ret.conn.Close() return nil, driver.ErrBadConn } return ret.conn, ret.err }
在看1.8 獲取連接的實現:
// Out of free connections or we were asked not to use one. If we're not // allowed to open any more connections, make a request and wait. if db.maxOpen > 0 && db.numOpen >= db.maxOpen { // Make the connRequest channel. It's buffered so that the // connectionOpener doesn't block while waiting for the req to be read. req := make(chan connRequest, 1) db.connRequests = append(db.connRequests, req) db.mu.Unlock() · // Timeout the connection request with the context. select { case <-ctx.Done(): return nil, ctx.Err() case ret, ok := <-req: if !ok { return nil, errDBClosed } if ret.err == nil && ret.conn.expired(lifetime) { ret.conn.Close() return nil, driver.ErrBadConn } return ret.conn, ret.err } }
很明顯看到在1.7里板惑,一旦connRequest的“消費者”過多橄镜,肯定有很多goroutine會一直等待的,no timeout冯乘,no cancel洽胶。connRequest的實現也是很有趣的,從命名上看是個conn的request裆馒,等待響應和拿到conn姊氓。connRequests是個chan的數組,而chan里面塞了conn喷好。當某goroutine消費完conn后翔横,變成生產者(參考putConnDBLocked方法),從而既能實現goroutine間的通知梗搅,同時又能直接傳遞數據庫連接禾唁。但是問題來了,channel本身是阻塞的些膨,也不支持超時蟀俊。一旦db出現慢查詢啥的,很可能就會導致大量數據庫請求阻塞订雾,蛋疼肢预。
golang的1.8很大程度上支持了context,從sql這個包里也能發(fā)現洼哎,新的大量的xxxContext函數出現了烫映。連接啊、執(zhí)行sql啊噩峦、事務啊锭沟。context.WithTimeout()直接就搞定超時功能了,煩惱不再识补。不但是database族淮,在1.7里net/http也早已大量支持了context∑就浚看來golang本身對context越來越倚重了祝辣,相信以后會看到越來越多的func(ctx context...)。