16 Scrapy爬取二級目錄

import scrapy
from ..items import SecondaryZufangItem

class ZufangSpider(scrapy.Spider):
    name = 'zufang'
    start_urls = ['http://bj.ganji.com/wblist/haidian/zufang/']
    #host_name = 'http://bj.ganji.com{}'

    def parse(self, response):
        zf = SecondaryZufangItem()
        next_link = response.xpath('//div[@class="pageBox"]/a[@class="next"]/@href').extract()[0]
        print(next_link,'*****'*20)
        yield scrapy.Request(next_link, callback = self.parse)

返回結(jié)果

SobersMacBookPro2016:secondary_zufang miraco$ scrapy crawl zufang
2018-12-20 14:58:24 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: secondary_zufang)
2018-12-20 14:58:24 [scrapy.utils.log] INFO: Versions: lxml 4.2.4.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.3 (v3.6.3:2c5fed86e0, Oct  3 2017, 00:32:08) - [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0i  14 Aug 2018), cryptography 2.3.1, Platform Darwin-17.7.0-x86_64-i386-64bit
2018-12-20 14:58:24 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'secondary_zufang', 'NEWSPIDER_MODULE': 'secondary_zufang.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['secondary_zufang.spiders']}
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled item pipelines:
['secondary_zufang.pipelines.SecondaryZufangPipeline']
2018-12-20 14:58:24 [scrapy.core.engine] INFO: Spider opened
2018-12-20 14:58:24 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-20 14:58:24 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-12-20 14:58:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/robots.txt> (referer: None)
2018-12-20 14:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/wblist/haidian/zufang/> (referer: None)
http://bj.ganji.com/haidian/zufang/pn2/ ****************************************************************************************************
2018-12-20 14:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn2/> (referer: http://bj.ganji.com/wblist/haidian/zufang/)
http://bj.ganji.com/haidian/zufang/pn3/ ****************************************************************************************************
2018-12-20 14:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn3/> (referer: http://bj.ganji.com/haidian/zufang/pn2/)
http://bj.ganji.com/haidian/zufang/pn4/ ****************************************************************************************************
2018-12-20 14:58:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn4/> (referer: http://bj.ganji.com/haidian/zufang/pn3/)
http://bj.ganji.com/haidian/zufang/pn5/ ****************************************************************************************************
2018-12-20 14:58:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn5/> (referer: http://bj.ganji.com/haidian/zufang/pn4/)
http://bj.ganji.com/haidian/zufang/pn6/ ****************************************************************************************************
2018-12-20 14:58:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn6/> (referer: http://bj.ganji.com/haidian/zufang/pn5/)
http://bj.ganji.com/haidian/zufang/pn7/ ****************************************************************************************************
2018-12-20 14:58:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn7/> (referer: http://bj.ganji.com/haidian/zufang/pn6/)
http://bj.ganji.com/haidian/zufang/pn8/ ****************************************************************************************************
2018-12-20 14:58:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn8/> (referer: http://bj.ganji.com/haidian/zufang/pn7/)
http://bj.ganji.com/haidian/zufang/pn9/ ****************************************************************************************************
2018-12-20 14:58:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn9/> (referer: http://bj.ganji.com/haidian/zufang/pn8/)
http://bj.ganji.com/haidian/zufang/pn10/ ****************************************************************************************************
2018-12-20 14:58:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn10/> (referer: http://bj.ganji.com/haidian/zufang/pn9/)
http://bj.ganji.com/haidian/zufang/pn11/ ****************************************************************************************************
2018-12-20 14:58:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn11/> (referer: http://bj.ganji.com/haidian/zufang/pn10/)
http://bj.ganji.com/haidian/zufang/pn12/ ****************************************************************************************************
2018-12-20 14:58:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn12/> (referer: http://bj.ganji.com/haidian/zufang/pn11/)
http://bj.ganji.com/haidian/zufang/pn13/ ****************************************************************************************************
2018-12-20 14:58:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn13/> (referer: http://bj.ganji.com/haidian/zufang/pn12/)
http://bj.ganji.com/haidian/zufang/pn14/ ****************************************************************************************************
2018-12-20 14:58:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn14/> (referer: http://bj.ganji.com/haidian/zufang/pn13/)
http://bj.ganji.com/haidian/zufang/pn15/ ****************************************************************************************************
2018-12-20 14:58:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn15/> (referer: http://bj.ganji.com/haidian/zufang/pn14/)
http://bj.ganji.com/haidian/zufang/pn16/ ****************************************************************************************************
2018-12-20 14:58:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn16/> (referer: http://bj.ganji.com/haidian/zufang/pn15/)
http://bj.ganji.com/haidian/zufang/pn17/ ****************************************************************************************************
2018-12-20 14:58:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn17/> (referer: http://bj.ganji.com/haidian/zufang/pn16/)
http://bj.ganji.com/haidian/zufang/pn18/ ****************************************************************************************************
2018-12-20 14:58:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn18/> (referer: http://bj.ganji.com/haidian/zufang/pn17/)
http://bj.ganji.com/haidian/zufang/pn19/ ****************************************************************************************************
2018-12-20 14:58:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn19/> (referer: http://bj.ganji.com/haidian/zufang/pn18/)
http://bj.ganji.com/haidian/zufang/pn20/ ****************************************************************************************************
2018-12-20 14:58:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn20/> (referer: http://bj.ganji.com/haidian/zufang/pn19/)
http://bj.ganji.com/haidian/zufang/pn21/ ****************************************************************************************************
2018-12-20 14:58:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn21/> (referer: http://bj.ganji.com/haidian/zufang/pn20/)
http://bj.ganji.com/haidian/zufang/pn22/ ****************************************************************************************************
2018-12-20 14:58:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn22/> (referer: http://bj.ganji.com/haidian/zufang/pn21/)
http://bj.ganji.com/haidian/zufang/pn23/ ****************************************************************************************************
2018-12-20 14:58:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn23/> (referer: http://bj.ganji.com/haidian/zufang/pn22/)
http://bj.ganji.com/haidian/zufang/pn24/ ****************************************************************************************************
2018-12-20 14:58:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn24/> (referer: http://bj.ganji.com/haidian/zufang/pn23/)
http://bj.ganji.com/haidian/zufang/pn25/ ****************************************************************************************************
2018-12-20 14:58:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn25/> (referer: http://bj.ganji.com/haidian/zufang/pn24/)
http://bj.ganji.com/haidian/zufang/pn26/ ****************************************************************************************************
2018-12-20 14:58:36 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://bj.ganji.com/haidian/zufang/pn26/> (referer: http://bj.ganji.com/haidian/zufang/pn25/)
2018-12-20 14:58:36 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://bj.ganji.com/haidian/zufang/pn26/>: HTTP status code is not handled or not allowed
2018-12-20 14:58:36 [scrapy.core.engine] INFO: Closing spider (finished)

是吧酬屉，要將獲取的url地址丟回調(diào)度器：

yield scrapy.Request(next_link[0], callback = self.parse)

這里不能寫return，因為還要接著爬呢匀伏。

import scrapy
from ..items import SecondaryZufangItem

class ZufangSpider(scrapy.Spider):
    name = 'zufang'
    start_urls = ['http://bj.ganji.com/wblist/haidian/zufang/']
    #host_name = 'http://bj.ganji.com{}'

    def parse(self, response):
        zf = SecondaryZufangItem()
        title_list = response.xpath('.//div[@class="f-list-item ershoufang-list"]//dd[@class="dd-item title"]/a/text()').extract()
        money_list = response.xpath('.//span[@class="num"]/text()').extract()

        for i,j in zip(title_list,money_list):
            zf['title']=i
            zf['money']=j
            yield zf

        next_link = response.xpath('//div[@class="pageBox"]/a[@class="next"]/@href').extract()


        if len(next_link) > 0:
            print(next_link,'*****'*20)
            yield scrapy.Request(next_link[0], callback = self.parse)
        else:
            pass #自行添加，一般是處理驗證頁面啥的

返回的挺多呢

image.png

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者

人面猴
序言：七十年代末，一起剝皮案震驚了整個濱河市单默，隨后出現(xiàn)的幾起案子，更是在濱河造成了極大的恐慌忘瓦，老刑警劉巖搁廓，帶你破解...
沈念sama閱讀 218,607評論 6贊 507
死咒
序言：濱河連續(xù)發(fā)生了三起死亡事件，死亡現(xiàn)場離奇詭異耕皮，居然都是意外死亡境蜕，警方通過查閱死者的電腦和手機(jī)，發(fā)現(xiàn)死者居然都...
沈念sama閱讀 93,239評論 3贊 395
救了他兩次的神仙讓他今天三更去死
文/潘曉璐我一進(jìn)店門凌停，熙熙樓的掌柜王于貴愁眉苦臉地迎上來粱年，“玉大人，你說我怎么就攤上這事罚拟√ㄊ” “怎么了完箩？”我有些...
開封第一講書人閱讀 164,960評論 0贊 355
道士緝兇錄：失蹤的賣姜人
文/不壞的土叔我叫張陵，是天一觀的道長拉庶。經(jīng)常有香客問我嗜憔，道長，這世上最難降的妖魔是什么氏仗？我笑而不...
開封第一講書人閱讀 58,750評論 1贊 294
?港島之戀（遺憾婚禮）
正文為了忘掉前任吉捶，我火速辦了婚禮，結(jié)果婚禮上皆尔，老公的妹妹穿的比我還像新娘呐舔。我一直安慰自己，他們只是感情好慷蠕，可當(dāng)我...
茶點故事閱讀 67,764評論 6贊 392
惡毒庶女頂嫁案：這布局不是一般人想出來的
文/花漫我一把揭開白布珊拼。她就那樣靜靜地躺著，像睡著了一般流炕。火紅的嫁衣襯著肌膚如雪觉既。梳的紋絲不亂的頭發(fā)上悴侵，一...
開封第一講書人閱讀 51,604評論 1贊 305
城市分裂傳說
那天，我揣著相機(jī)與錄音，去河邊找鬼诊沪。笑死哥攘，一個胖子當(dāng)著我的面吹牛虱颗，可吹牛的內(nèi)容都是我干的罗丰。我是一名探鬼主播，決...
沈念sama閱讀 40,347評論 3贊 418
雙鴛鴦連環(huán)套：你想象不到人心有多黑
文/蒼蘭香墨我猛地睜開眼挠将，長吁一口氣：“原來是場噩夢啊……” “哼胳岂！你這毒婦竟也來了？” 一聲冷哼從身側(cè)響起舔稀，我...
開封第一講書人閱讀 39,253評論 0贊 276
萬榮殺人案實錄
序言：老撾萬榮一對情侶失蹤乳丰，失蹤者是張志新（化名）和其女友劉穎，沒想到半個月后内贮，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體产园，經(jīng)...
沈念sama閱讀 45,702評論 1贊 315
?護(hù)林員之死
正文獨居荒郊野嶺守林人離奇死亡，尸身上長有42處帶血的膿包…… 初始之章·張勛以下內(nèi)容為張勛視角年9月15日...
茶點故事閱讀 37,893評論 3贊 336
?白月光啟示錄
正文我和宋清朗相戀三年贺归，在試婚紗的時候發(fā)現(xiàn)自己被綠了淆两。大學(xué)時的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
茶點故事閱讀 40,015評論 1贊 348
活死人
序言：一個原本活蹦亂跳的男人離奇死亡拂酣，死狀恐怖秋冰，靈堂內(nèi)的尸體忽然破棺而出，到底是詐尸還是另有隱情婶熬，我是刑警寧澤剑勾，帶...
沈念sama閱讀 35,734評論 5贊 346
?日本核電站爆炸內(nèi)幕
正文年R本政府宣布埃撵，位于F島的核電站，受9級特大地震影響虽另，放射性物質(zhì)發(fā)生泄漏暂刘。R本人自食惡果不足惜，卻給世界環(huán)境...
茶點故事閱讀 41,352評論 3贊 330
男人毒藥：我在死后第九天來索命
文/蒙蒙一捂刺、第九天我趴在偏房一處隱蔽的房頂上張望谣拣。院中可真熱鬧，春花似錦族展、人聲如沸森缠。這莊子的主人今日做“春日...
開封第一講書人閱讀 31,934評論 0贊 22
一樁弒父案仪缸，背后竟有這般陰謀
文/蒼蘭香墨我抬頭看了看天上的太陽贵涵。三九已至，卻和暖如春恰画，著一層夾襖步出監(jiān)牢的瞬間宾茂，已是汗流浹背。一陣腳步聲響...
開封第一講書人閱讀 33,052評論 1贊 270
情欲美人皮
我被黑心中介騙來泰國打工拴还，沒想到剛下飛機(jī)就差點兒被人妖公主榨干…… 1. 我叫王不留跨晴，地道東北人。一個月前我還...
沈念sama閱讀 48,216評論 3贊 371
代替公主和親
正文我出身青樓自沧，卻偏偏與公主長得像坟奥，于是被迫代替她去往敵國和親树瞭。傳聞我的和親對象是個殘疾皇子拇厢，可洞房花燭夜當(dāng)晚...
茶點故事閱讀 44,969評論 2贊 355

16 Scrapy爬取二級目錄

推薦閱讀更多精彩內(nèi)容