title: Python 讀寫(xiě) hbase 數(shù)據(jù)的正確姿勢(shì)（三）
tags:
- hbase
- happybase
- python
categories:
- ?Hbase
comments: true
date: 2017-09-28 23:00:00
問(wèn)題3: 查詢異常 [Errno 32] Broken pipe
這篇文章將繼續(xù) python-hbase 這一話題张症，討論一個(gè)在線上環(huán)境中出現(xiàn)的很有意思的一個(gè)問(wèn)題应媚。
問(wèn)題描述
同樣是前面文章中描述的類似查詢場(chǎng)景，生產(chǎn)測(cè)試彰阴，在調(diào)用 hbase thrift 接口時(shí)，日志中捕獲到大量的 [Errno 32] Broken pipe 錯(cuò)誤鸠天，具體如下：
File "/usr/local/lib/python2.7/site-packages/happybase/table.py", line 402, in scan
self.name, scan, {})
File "/usr/local/lib/python2.7/site-packages/thriftpy/thrift.py", line 195, in _req
self._send(_api, **kwargs)
File "/usr/local/lib/python2.7/site-packages/thriftpy/thrift.py", line 206, in _send
self._oprot.write_message_end()
File "thriftpy/protocol/cybin/cybin.pyx", line 463, in cybin.TCyBinaryProtocol.write_message_end (thriftpy/protocol/cybin/cybin.c:6845)
File "thriftpy/transport/buffered/cybuffered.pyx", line 80, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.c_flush (thriftpy/transport/buffered/cybuffered.c:2147)
File "/usr/local/lib/python2.7/site-packages/thriftpy/transport/socket.py", line 129, in write
self.sock.sendall(buff)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
問(wèn)題分析
[Errno 32] Broken pipe 往往意味著是連接問(wèn)題烁涌，連接的一端已經(jīng)關(guān)閉連接，但是另一端仍然使用這個(gè)連接向?qū)Ψ桨l(fā)送數(shù)據(jù)牺蹄。
經(jīng)過(guò)和平臺(tái)組同學(xué)的排查測(cè)試掐隐，排除了生產(chǎn)環(huán)境中鏈路質(zhì)量的問(wèn)題以及 hbase thrift server 穩(wěn)定性的問(wèn)題。如果不是網(wǎng)絡(luò)問(wèn)題導(dǎo)致，那有沒(méi)有可能是 hbase 處理請(qǐng)求的過(guò)程中發(fā)生了錯(cuò)誤虑省，主動(dòng)關(guān)閉了連接匿刮？為了進(jìn)一步追查問(wèn)題，在本地還原了線上場(chǎng)景探颈，復(fù)現(xiàn)錯(cuò)誤熟丸。
果不其然正压，還原現(xiàn)場(chǎng)后客戶端最早收到了一個(gè)額外的異常加袋，這之后才收到大量的 Broken Pipe 錯(cuò)誤一也，新的異常：
File "/usr/local/lib/python2.7/site-packages/happybase/table.py", line 402, in scan
self.name, scan, {})
File "/usr/local/lib/python2.7/site-packages/thriftpy/thrift.py", line 198, in _req
return self._recv(_api)
File "/usr/local/lib/python2.7/site-packages/thriftpy/thrift.py", line 210, in _recv
fname, mtype, rseqid = self._iprot.read_message_begin()
File "thriftpy/protocol/cybin/cybin.pyx", line 429, in cybin.TCyBinaryProtocol.read_message_begin (thriftpy/protocol/cybin/cybin.c:6325)
File "thriftpy/protocol/cybin/cybin.pyx", line 60, in cybin.read_i32 (thriftpy/protocol/cybin/cybin.c:1546)
File "thriftpy/transport/buffered/cybuffered.pyx", line 65, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.c_read (thriftpy/transport/buffered/cybuffered.c:1881)
File "thriftpy/transport/buffered/cybuffered.pyx", line 69, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.read_trans (thriftpy/transport/buffered/cybuffered.c:1948)
File "thriftpy/transport/cybase.pyx", line 61, in thriftpy.transport.cybase.TCyBuffer.read_trans (thriftpy/transport/cybase.c:1472)
File "/usr/local/lib/python2.7/site-packages/thriftpy/transport/socket.py", line 125, in read
message='TSocket read 0 bytes')
TTransportException: TTransportException(message='TSocket read 0 bytes', type=4)
這個(gè)異常的大概意思是 server 端發(fā)生了異常并沒(méi)有返回任何數(shù)據(jù)桅打，扒一下 hbase server 的日志命迈，又發(fā)現(xiàn)了一個(gè)有趣的異常：
java.lang.IllegalArgumentException: Incorrect Filter String
at org.apache.hadoop.hbase.filter.ParseFilter.extractFilterSimpleExpression(ParseFilter.java:226)
at org.apache.hadoop.hbase.filter.ParseFilter.parseFilterString(ParseFilter.java:174)
at org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerOpenWithScan(ThriftServerRunner.java:1481)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)
at com.sun.proxy.$Proxy9.scannerOpenWithScan(Unknown Source)
at org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4613)
at org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4597)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
這個(gè)異常中出現(xiàn)了解決問(wèn)題的核心關(guān)鍵詞 Incorrect Filter String肤粱，從字面理解來(lái)看是傳到 hbase 的 filter 不正確因俐，導(dǎo)致解析失敗精肃。
找到這條有問(wèn)題的 filter：
"SingleColumnValueFilter('basic', 'ArticleTypeID', =, 'binary:\x00\x00\x00\x00\x03'T\xc9')"
從語(yǔ)法上來(lái)看感覺(jué)并沒(méi)有什么問(wèn)題化借，同樣的邏輯使用 java client 則完全正確潜慎。為了一探究竟拉出 hbase 拋錯(cuò)部分的源碼看看為什么：
public byte [] extractFilterSimpleExpression (byte [] filterStringAsByteArray,
int filterExpressionStartOffset)
throws CharacterCodingException {
int quoteCount = 0;
for (int i=filterExpressionStartOffset; i<filterStringAsByteArray.length; i++) {
if (filterStringAsByteArray[i] == ParseConstants.SINGLE_QUOTE) {
if (isQuoteUnescaped(filterStringAsByteArray, i)) {
quoteCount ++;
} else {
// To skip the next quote that has been escaped
i++;
}
}
if (filterStringAsByteArray[i] == ParseConstants.RPAREN && (quoteCount %2 ) == 0) {
byte [] filterSimpleExpression = new byte [i - filterExpressionStartOffset + 1];
Bytes.putBytes(filterSimpleExpression, 0, filterStringAsByteArray,
filterExpressionStartOffset, i-filterExpressionStartOffset + 1);
return filterSimpleExpression;
}
}
throw new IllegalArgumentException("Incorrect Filter String");
}
從源碼中可以看到在兩種情況下會(huì) raise exception：
- filter 不是以
ParseConstants.RPAREN結(jié)尾，即不是以)結(jié)尾 -
quoteCount不是偶數(shù)蓖康，即單引號(hào)的數(shù)量不是偶數(shù)
到此铐炫，可以真相大白了，從代碼中可以看到 hbase parse 的過(guò)程中是通過(guò)單引號(hào)提取參數(shù)的蒜焊，而我的 filter 中有一個(gè)整型參數(shù)在轉(zhuǎn)成 bytes 后包含單引號(hào)倒信，影響了 hbase ?解析 filter 參數(shù)，并最終導(dǎo)致 quoteCount 不是偶數(shù)泳梆，然后拋出異常鳖悠。
解決問(wèn)題
定位到問(wèn)題根本后，解決問(wèn)題就 so easy 了优妙。解析 filter 的源碼中用到了 escape quote 的方法 isQuoteUnescaped竞穷，具體實(shí)現(xiàn)如下：
public static boolean isQuoteUnescaped (byte [] array, int quoteIndex) {
if (array == null) {
throw new IllegalArgumentException("isQuoteUnescaped called with a null array");
}
if (quoteIndex == array.length - 1 || array[quoteIndex+1] != ParseConstants.SINGLE_QUOTE) {
return true;
}
else {
return false;
}
}
邏輯很簡(jiǎn)單，判斷單引號(hào)下一下字符是否仍然是單引號(hào)鳞溉，如果是則被轉(zhuǎn)義瘾带，跳過(guò)檢查。所以我們的filter 只需要通過(guò) 兩個(gè)單引號(hào) 替換參數(shù)中的 一個(gè)單引號(hào)即可熟菲，eg:
hbase_int_filter_template = "SingleColumnValueFilter('a', '{property}', {symbol}, 'binary:{threshold}')"
f = hbase_int_filter_template.format(property=params[0], symbol=flag, threshold=struct.pack('>q', threshold).replace("'", "''"))
單引號(hào)轉(zhuǎn)義后看政，沒(méi)有再出現(xiàn)這兩類 exception。
后續(xù)分析
回頭來(lái)看抄罕，因?yàn)?filter 使用錯(cuò)誤允蚣，導(dǎo)致 hbase 解析 filter 異常，hbase server 拋出異常呆贿，并中斷連接嚷兔，client 收到 TTransportException 異常森渐，此時(shí)這條連接已經(jīng)失效，但是仍在 connection pool 中冒晰，所有后續(xù)從 connection pool 中獲取連接拿到這條連接后同衣，再向 hbase 發(fā)送請(qǐng)求時(shí) client 端不斷收到 [Errno 32] Broken pipe 錯(cuò)誤。
出現(xiàn)了一個(gè)新的思考題壶运，為什么一條異常的連接會(huì)出現(xiàn)在 connect pool 中耐齐，而且總會(huì)拿到這條連接 ？
題外話
仔細(xì)想想 filter 中單引號(hào)需要轉(zhuǎn)義這種情況按理說(shuō) hbase 會(huì)在官方的 Document 中提到才對(duì)蒋情，翻翻 user guide 埠况，果然找到了 Filter Language
使用前用心看看官方文檔還是很有必要的，可以少踩許多坑...
小結(jié)
對(duì)于使用 thrift 接口查詢 hbase 的場(chǎng)景棵癣，使用 filter 時(shí)要注意將 filter str 中函數(shù)參數(shù)中的單引號(hào)轉(zhuǎn)義辕翰，否則 hbase 無(wú)法正確解析 filter，導(dǎo)致 TTransportException狈谊。
