通過(guò)對(duì)廖雪峰的python教程學(xué)習(xí)生成器啃勉，如下代碼：
def odd():
print('step 1')
yield 1
print('step 2')
yield (3)
print('step 3')
yield (5)
if __name__ == "__main__":
o = odd()
for index in o:
print(index)
輸出內(nèi)容如下：
step 1
1
step 2
3
step 3
5
generator函數(shù)罕拂，在每次調(diào)用next()的時(shí)候執(zhí)行仰剿，遇到y(tǒng)ield語(yǔ)句返回卧须，再次執(zhí)行時(shí)從上次返回的yield語(yǔ)句處繼續(xù)執(zhí)行另绩。for循環(huán)就是調(diào)用next()函數(shù)，理解了這點(diǎn)就可以理解上述代碼了花嘶。再scrapy框架爬蟲(chóng)中經(jīng)常會(huì)遇到y(tǒng)ield函數(shù)笋籽，
def start_requests(self):
self.log('------' + __name__ + ' start requests ------')
if self.task_running is False:
return
apps = appinfo_mq.query_star_ids(self.market, self.country, self.start_id,
self.start_index, self.keyword_count - self.start_index)
header = CommentsSpider.headers
# apps = ['548984223'] #文件管理器
if apps is not None:
log_file = open(self.log_path, 'a')
for app in apps:
app = app.replace('id', '')
log_file.write(str(app) + '---')
self.page_index[str(app)] = 1
self.is_first[str(app)] = True
new_url = CommentsSpider.url.format(app, 1)
yield Request(new_url, headers=header, meta={'app_id': app})
log_file.close()
else:
yield None
調(diào)用如下：
for req in self.start_requests():
if req is not None:
self.crawler.engine.crawl(req, spider=self)
self.no_keyword = False
else:
self.task_running = False
self.no_keyword = True
timer.check_keyword_recover(self.request_action)
break
我們的start_requests()函數(shù)生成一個(gè)generator，通過(guò)循環(huán)逐一拿到Request（）請(qǐng)求椭员，
通過(guò)我們的引擎self.crawler.engine對(duì)每一個(gè)網(wǎng)絡(luò)請(qǐng)求進(jìn)行爬取车海，
Request（）是scrapy內(nèi)部封裝的網(wǎng)絡(luò)請(qǐng)求。我們?cè)谂老x(chóng)中將所有的請(qǐng)求放入generator隘击，
后面通過(guò)generator來(lái)靈活處理我們的請(qǐng)求侍芝。
