思路：
- step 1：爬取單個(gè)詳情頁(yè)的信息（getInfo(url)函數(shù)）
- step 2：在一個(gè)列表頁(yè)爬取所有詳情頁(yè)的鏈接驯绎，再調(diào)用getInfo(url)函數(shù)有决，得到該列表頁(yè)中所有詳情頁(yè)的信息（onePageInfo(list_url)函數(shù)）
- step 3：通過(guò)觀察每個(gè)列表頁(yè)網(wǎng)址秉撇，發(fā)現(xiàn)網(wǎng)址組成規(guī)律灵迫，調(diào)用onePageInfo(list_url)爬取信息
難點(diǎn)：
判斷房東性別
結(jié)果展示：
總結(jié)：
- 房東性別通過(guò)房東圖片右下角的小圖片來(lái)判斷照雁，小圖片class屬性能區(qū)分男女蚕愤，存在的問(wèn)題是有些房東沒(méi)設(shè)置性別，無(wú)法判斷饺蚊，這里默認(rèn)為男性萍诱。通過(guò)字符串format()方法嵌套if-else實(shí)現(xiàn)
- 發(fā)現(xiàn)得到的address后面跟了\n和一段空格，這里用字符串rstrip()方法去除
- 房間圖片的屬性為style污呼，這里用列表切片得到需要信息
- 列表推導(dǎo)式：
urls = ['{}'.format(str(i)) for i in range(1,8)] 正確
urls = ['{}'.format(str(i) for i in range(1,8))] 錯(cuò)誤
我的代碼：
frombs4importBeautifulSoup
importrequests
#獲得單個(gè)詳情頁(yè)的信息
defgetInfo(url):
wb_data = requests.get(url)
soup = BeautifulSoup(wb_data.text,'lxml')
titles = soup.select('body > div.wrap.clearfix.con_bg > div.con_l > div.pho_info > h4 > em')
addresses = soup.select('body > div.wrap.clearfix.con_bg > div.con_l > div.pho_info > p > span')
rentals = soup.select('#pricePart > div.day_l > span')
house_images = soup.select('#imgMouseCusor')
lord_images = soup.select('#floatRightBox > div.js_box.clearfix > div.member_pic > a > img')
lord_names = soup.select('#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > a')
lord_sexs = soup.select('#floatRightBox > div.js_box.clearfix > div.member_pic > div')
fortitle,address,rental,house_image,lord_image,lord_name,lord_sexinzip(titles,addresses,rentals,house_images,lord_images,lord_names,lord_sexs):
data = {
'title': title.get_text(),
'address': address.get_text().rstrip(),
'rental': rental.get_text(),
'house_image': house_image.get('style')[16:-2],
'lord_image': lord_image.get('src'),
'lord_name': lord_name.get_text(),
'lord_sex':"{}".format('female'iflord_sex['class'][0] =="member_ico1"else'male')
}
print(data)
#得到一整頁(yè)
defonePageInfo(list_url):
page_list = requests.get(list_url)
page_list_soup = BeautifulSoup(page_list.text,'lxml')
house_list = page_list_soup.select('#page_list > ul > li > a')
'''
house_link = []
for i in house_list:
house_link.append(i['href'])
print(house_link)
'''
foriinhouse_list:
getInfo(i['href'])
urls = ['http://bj.xiaozhu.com/search-duanzufang-p{}-0/'.format(str(i))foriinrange(1,8)]
#urls = ['http://bj.xiaozhu.com/search-duanzufang-p{}-0/'.format(str(i) for i in range(1,8))]錯(cuò)誤
forurlinurls:
onePageInfo(url)
