pyspider代碼結(jié)構(gòu)
主要從以下幾個(gè)模塊看pyspider源碼
- libs里面的工具類五慈。比如最常用的basehandler 等等。
- process依溯，scheduler毯炮，resultdb。其中最容易看得是resultdb
- webui部分肃弟，web如何調(diào)用數(shù)據(jù)庫(kù)的腳本等
debug的一些技巧
pyspider在run.py啟動(dòng)了多個(gè)線程玷室， 多個(gè)線程的debug其實(shí)很麻煩，我們需要根據(jù)上一個(gè)欄目的代碼結(jié)構(gòu)來一個(gè)個(gè)的看笤受。
比如最簡(jiǎn)單的resultdb模塊穷缤，只有一個(gè)result_worker.py文件，看懂了這個(gè)文件箩兽，其實(shí)就看懂了這個(gè)模塊津肛。
class ResultWorker(object):
"""
do with result
override this if needed.
"""
def __init__(self, resultdb, inqueue):
self.resultdb = resultdb
self.inqueue = inqueue
self._quit = False
def on_result(self, task, result):
'''Called every result'''
if not result:
return
if 'taskid' in task and 'project' in task and 'url' in task:
logger.info('result %s:%s %s -> %.30r' % (
task['project'], task['taskid'], task['url'], result))
return self.resultdb.save(
project=task['project'],
taskid=task['taskid'],
url=task['url'],
result=result
)
else:
logger.warning('result UNKNOW -> %.30r' % result)
return
def quit(self):
self._quit = True
def run(self):
'''Run loop'''
logger.info("result_worker starting...")
while not self._quit:
try:
task, result = self.inqueue.get(timeout=1)
self.on_result(task, result)
except Queue.Empty as e:
continue
except KeyboardInterrupt:
break
except AssertionError as e:
logger.error(e)
continue
except Exception as e:
logger.exception(e)
continue
logger.info("result_worker exiting...")
class OneResultWorker(ResultWorker):
'''Result Worker for one mode, write results to stdout'''
def on_result(self, task, result):
'''Called every result'''
if not result:
return
if 'taskid' in task and 'project' in task and 'url' in task:
logger.info('result %s:%s %s -> %.30r' % (
task['project'], task['taskid'], task['url'], result))
print(json.dumps({
'taskid': task['taskid'],
'project': task['project'],
'url': task['url'],
'result': result,
'updatetime': time.time()
}))
else:
logger.warning('result UNKNOW -> %.30r' % result)
return
OneResultWorker不需要看，功能上汗贫，這個(gè)模塊和上面實(shí)現(xiàn)一樣的功能身坐，應(yīng)該是用來單獨(dú)啟動(dòng)的時(shí)候調(diào)用的代碼。同樣的道理落包，很多one開頭的代碼都可以掠過部蛇。
