之前一段時(shí)間把畢業(yè)論文搞完了莱褒。幫學(xué)弟對(duì)旅游景點(diǎn)的數(shù)據(jù)進(jìn)行可視化的展示
一、數(shù)據(jù)準(zhǔn)備
這里使用的是去哪兒的旅游數(shù)據(jù)，參考的是大吉大利小米醬的簡(jiǎn)書(shū)http://www.reibang.com/p/b7627e67b6b9有興趣的可以去看看蟹倾。大吉大利小米醬
簡(jiǎn)書(shū)寫(xiě)的俏皮風(fēng)趣（吾輩之楷模）。
import time,requests, re
from lxml import etree
import pandas as pd
def getPage(url):
headers = {
'Accept':'application/json, text/javascript, */*; q=0.01',
'Accept-Encoding':'gzip, deflate',
'Accept-Language':'zh-CN,zh;q=0.9,en;q=0.8',
'Connection':'keep-alive',
'cookie':'QN1=dXrghVzaM0cN+wYpRryRAg==; QN269=9B0D2CC075F511E9BCA9FA163ED025F3; _i=ueHd8xMxBTVXqvCA4qR470Um5eyX; QunarGlobal=10.86.213.150_21340582_16ab448be93_7a80|1557803849001; QN99=647; fid=eda14267-76af-406d-975d-9e098c9d50f4; __utma=183398822.428737805.1557803850.1557803850.1557803850.1; __utmz=183398822.1557803850.1.1.utmcsr=baidu|utmccn=(organic)|utmcmd=organic|utmctr=%E5%A5%BD123%E6%97%85%E6%B8%B8; _jzqa=1.1924495795568354000.1557803852.1557803852.1557803852.1; _jzqx=1.1557803852.1557803852.1.jzqsr=hotel%2Equnar%2Ecom|jzqct=/.-; QN71="NTguMjQzLjI1NC4xMjk65a6J5b69OjE="; QN57=15578038546700.26539516250028505; QN67=1211; cto_lwid=06e141f7-870f-42ca-9519-312c3587bed8; QN63=%E7%83%AD%E9%97%A8%E6%99%AF%E7%82%B9%7C%E5%AE%89%E5%BE%BD; csrfToken=ZrtgmFh4YnLvKzqP8dCzzti2JoLh7sFz; _vi=3di8PYGZ-R9GtBBMOJN2tlBVR_KX-2jyHo6YmCS3JC-NB1B9rZnnrT7ZlQIEIdYWcInEmtNp0xdr9Aaf4mU9WCoN7q_-8QEnQ3m2goXxwFZpqYH8gGaupHTLFP-ak8JMAES3K_yVSwvsNpRb_qYHXgc4xajzIvF93Hnj_YkbVQDN; Hm_lvt_15577700f8ecddb1a927813c81166ade=1557803855,1558319263; QN300=auto_4e0d874a; QN205=auto_4e0d874a; QN277=auto_4e0d874a; QN267=79056878cd9f7ec1; QN58=1558319262033%7C1558319518464%7C2; Hm_lpvt_15577700f8ecddb1a927813c81166ade=1558319519; JSESSIONID=A509706026389E421E058210F3BEF192; Request-Node=f6769d42991e67d1c5a98893de33d1f3; QN271=06f7c456-ac81-4bdb-b833-ddfbc81ab633',
'Host':'piao.qunar.com',
'Referer':'http://piao.qunar.com/ticket/list.htm?keyword=%E7%83%AD%E9%97%'
'A8%E6%99%AF%E7%82%B9®ion=&from=mpl_search_suggest',
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36',
'X-Requested-With':'XMLHttpRequest'
}
try:
page = requests.get(url,headers = headers)
print("page:")
print(page)
return page
except Exception as e:
print(str(e))
def getList():
place = '熱門(mén)景點(diǎn)'
url = 'http://piao.qunar.com/ticket/list.htm?keyword='+ place +'®ion=&from=mpl_search_suggest&page={}'
i = 1
sightlist = []
while i < 3000:
page = getPage(url.format(i))
selector = etree.HTML(page.text)
print('正在爬取第', str(i), '頁(yè)景點(diǎn)信息')
i += 1
informations = selector.xpath('//div[@class="result_list"]/div')
for inf in informations:
try:
sight_name = inf.xpath('./div/div/h3/a/text()')[0]
sight_level = inf.xpath('.//span[@class="level"]/text()')
if len(sight_level):
sight_level = sight_level[0].replace('景區(qū)','')
else:
sight_level = 0
sight_area = inf.xpath('.//span[@class="area"]/a/text()')[0]
sight_hot = inf.xpath('.//span[@class="product_star_level"]//span/text()')[0].replace('熱度 ','')
sight_add = inf.xpath('.//p[@class="address color999"]/span/text()')[0]
sight_add = re.sub('地址：|（.*?）|\(.*?\)|猖闪，.*?$|\/.*?$','',str(sight_add))
sight_slogen = inf.xpath('.//div[@class="intro color999"]/text()')[0]
sight_price = inf.xpath('.//span[@class="sight_item_price"]/em/text()')
if len(sight_price):
sight_price = sight_price[0]
else:
i = 0
break
sight_soldnum = inf.xpath('.//span[@class="hot_num"]/text()')[0]
sight_point = inf.xpath('./@data-point')[0]
sight_url = inf.xpath('.//h3/a[@class="name"]/@href')[0]
sightlist.append([sight_name,sight_level,sight_area,float(sight_price),int(sight_soldnum),float(sight_hot),sight_add.replace('地址：',''),sight_point,sight_slogen,sight_url])
print("sightlist:")
print(sightlist)
except IndexError:
pass
time.sleep(10)
return sightlist
def listToExcel(list,name):
df = pd.DataFrame(list,columns=['景點(diǎn)名稱(chēng)','級(jí)別','所在區(qū)域','起步價(jià)','銷(xiāo)售量','熱度','地址','經(jīng)緯度','標(biāo)語(yǔ)','詳情網(wǎng)址'])
df.to_csv(name + ".csv", sep=',')
def main():
sightlist = getList()
listToExcel(sightlist,'hotplace')
if __name__=='__main__':
main()
在pycharm中運(yùn)行代碼鲜棠。將去哪兒網(wǎng)中熱門(mén)的景點(diǎn)數(shù)據(jù)進(jìn)行爬去保存為.csv數(shù)據(jù)，包括景點(diǎn)名稱(chēng)培慌、級(jí)別豁陆、所在地區(qū)等信息。
熱門(mén)景點(diǎn)數(shù)據(jù)
但是這些數(shù)據(jù)并不能進(jìn)行直接的可視化（這里小米醬將爬去的數(shù)據(jù)用python轉(zhuǎn)換成json文件吵护，用echart進(jìn)行可視化）盒音。
小米醬的可視化
二、數(shù)據(jù)處理工作
為了更好的將數(shù)據(jù)進(jìn)行可視化馅而。我們把csv數(shù)據(jù)做了分列處理祥诽，將經(jīng)緯度分為兩行。
經(jīng)緯度分列
分列后
然后將數(shù)據(jù)導(dǎo)入ArcMap中瓮恭，在ArcMap中將景點(diǎn)數(shù)據(jù)轉(zhuǎn)化為shp格式的點(diǎn)數(shù)據(jù)雄坪。
i顯示X、Y數(shù)據(jù)
點(diǎn)擊按X屯蹦、Y顯示數(shù)據(jù)后维哈，X绳姨、Y字段和輸入坐標(biāo)的坐標(biāo)系（我選的是WGS84）
設(shè)置參數(shù)
點(diǎn)擊確定，就可以將數(shù)據(jù)轉(zhuǎn)換為shp格式的point數(shù)據(jù)（經(jīng)緯度坐標(biāo)值不能搞錯(cuò)了）阔挠。為了確保沒(méi)有錯(cuò)飘庄，最好加載在線的地圖確認(rèn)數(shù)據(jù)轉(zhuǎn)換的正確性。
經(jīng)緯度數(shù)據(jù)錯(cuò)誤效果
數(shù)據(jù)轉(zhuǎn)換
（POI數(shù)據(jù)也可以通過(guò)此方法轉(zhuǎn)換為shp數(shù)據(jù)）然后將數(shù)據(jù)導(dǎo)出為shp即可谒亦。
三竭宰、數(shù)據(jù)可視化
本次將數(shù)據(jù)轉(zhuǎn)化為GeoJSON數(shù)據(jù)，通過(guò)leaflets的API進(jìn)行展示份招。包括熱力圖切揭、彈窗、標(biāo)注锁摔、定位搜索廓旬。
代碼界面
可視化效果
整個(gè)Demo的代碼放在Github上（老鐵們可以把Github上的小星星點(diǎn)一點(diǎn)）：https://github.com/hunim/Tourism-data-visualization
