來源：KOTO - kesci.com
原文鏈接：丁香園--新型冠狀病毒肺炎論壇的數(shù)據(jù)爬取和分析
點(diǎn)擊以上鏈接?? 不用配置環(huán)境晴圾，直接在線運(yùn)行

DXY.com丁香園是國內(nèi)最大的醫(yī)學(xué)綜合網(wǎng)站之一机隙，其醫(yī)學(xué)論壇聚集了國內(nèi)的專業(yè)人士。本文選取其中的論壇（新型冠狀病毒肺炎）抓取數(shù)據(jù)來分析人們對(duì)于疫情的討論關(guān)注

一、獲取數(shù)據(jù)

從dxy.com獲取數(shù)據(jù)

import pandas as pd
import numpy as np
import re
import urllib.request
from bs4 import BeautifulSoup
from tqdm import tqdm 
from collections import Counter
import matplotlib.pyplot as plt

# url="http://www.dxy.cn/bbs/board/288?age=30&tpg=1"
# req= urllib.request.urlopen(url)
# soup=BeautifulSoup(req,"html.parser")
# print(soup)
info_1=[]
info_2=[]
info_3=[]
info_4=[]
info_5=[]
info_6=[]
info_7=[]
with tqdm(range(1,28)) as t:
    for j in t:
        url="http://www.dxy.cn/bbs/board/288?age=30&tpg=%a"%j
        req=urllib.request.urlopen(url)
        soup=BeautifulSoup(req,"html.parser")
        links=soup.find_all("td",attrs={"class":"news"})
        # print(len(links))
        for i in range(len(links)):
            title=soup.select('#col-1 > table.post-table > tbody > tr:nth-child('+str(i+1)+') > td.news > a')
            a=title[0].get_text()
            info_1.append(a)
            
            author=soup.select('#col-1 > table.post-table > tbody > tr:nth-child('+str(i+1)+') > td:nth-child(3) > a')
            a=author[0].get_text()
            info_2.append(a)
            
            first_time=soup.select('#col-1 > table.post-table > tbody > tr:nth-child('+str(i+1)+') > td:nth-child(3) > em')
            a=first_time[0].get_text()
            info_3.append(a)
            
            reply_num=soup.select('#col-1 > table.post-table > tbody > tr:nth-child('+str(i+1)+') > td.num.calign > a')
            a=reply_num[0].get_text()
            info_4.append(a)
            
            click_num=soup.select('#col-1 > table.post-table > tbody > tr:nth-child('+str(i+1)+') > td.num.calign > em')
            a=click_num[0].get_text()
            info_5.append(a)
            
            last_replaier=soup.select('#col-1 > table.post-table > tbody > tr:nth-child('+str(i+1)+') > td.by.ralign.last > a')
            a=last_replaier[0].get_text()
            info_6.append(a)
            
            last_time=soup.select('#col-1 > table.post-table > tbody > tr:nth-child('+str(i+1)+') > td.by.ralign.last > em')
            a=last_time[0].get_text()
            info_7.append(a)
            
t.close()
df=pd.DataFrame({'content':info_1,'author':info_2,'post_time':info_3,'reply_num':info_4,
'click_num':info_5,'last_reply':info_6,'reply_time':info_7})
df.to_csv("forum.csv",index=True,sep=',')
    

df.head()

二邓深、數(shù)據(jù)清洗

df.isna().any()  # 查空值

content       False
author        False
post_time     False
reply_num     False
click_num     False
last_reply    False
reply_time    False
dtype: bool

df[df.isna().values == True]  # 檢查具體缺失值的數(shù)據(jù)

df = df.dropna(axis=0)
df.isna().any()  # 查空值

content       False
author        False
post_time     False
reply_num     False
click_num     False
last_reply    False
reply_time    False
dtype: bool

df.duplicated().any()  # 查詢有無重復(fù)值

False

df[df.duplicated().values == True]  # 查詢具體重復(fù)數(shù)據(jù)

df = df.drop_duplicates()  # 去重復(fù)值
df.duplicated().any()  # 再次檢查確認(rèn)有無重復(fù)

False

df.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 943 entries, 0 to 942
Data columns (total 7 columns):
content       943 non-null object
author        943 non-null object
post_time     943 non-null object
reply_num     943 non-null object
click_num     943 non-null object
last_reply    943 non-null object
reply_time    943 non-null object
dtypes: object(7)
memory usage: 58.9+ KB

三浓冒、分析處理數(shù)據(jù)

df['reply_num']=df['reply_num'].astype('float')
df['reply_num'].describe()  # 查看帖子回復(fù)情況的統(tǒng)計(jì)值

count     943.000000
mean       24.067869
std        90.467069
min         0.000000
25%         0.000000
50%         2.000000
75%        10.000000
max      1953.000000
Name: reply_num, dtype: float64

回復(fù)最多的帖子有1956次，平均回復(fù)12.025668次所森。

查看回復(fù)最多的100條帖子

df_re = df.sort_values('reply_num', ascending=False).iloc[:100, :]  # 按回復(fù)降序排列
df_re.head()

前100熱門帖子有無同一個(gè)人發(fā)表的囱持，顯示是有的

df_re['author'].duplicated().any()  # 前100熱門帖子有無同一個(gè)人發(fā)表的

True

df_re[df_re['author'].duplicated()].count()

content       65
author        65
post_time     65
reply_num     65
click_num     65
last_reply    65
reply_time    65
dtype: int64

Counter(df_re['author']).most_common(5)  # 在回復(fù)最多的前100個(gè)帖子中，發(fā)表數(shù)量前幾位的作者焕济，估計(jì)其中資深專家

[('lightningwing', 21),
 ('丁香調(diào)查官方賬號(hào)', 11),
 ('阿呆233', 7),
 ('DR的理想', 6),
 ('安第斯杰克', 5)]

下面我再試試用時(shí)間排序可否找出一些論壇發(fā)展情況纷妆。

按照時(shí)間排序

df['counts'] = 1  # 插入一列用來計(jì)數(shù)
df.head()

df_dateindex = df.sort_values('post_time')  # 設(shè)置按發(fā)表日期排序

df_dateindex.index = pd.to_datetime(df_dateindex['post_time'])  # 設(shè)置發(fā)表日期為index

df_resample_Y = df_dateindex.resample('1D')  # 按天分類統(tǒng)計(jì)
df_resample_Y = df_resample_Y

count_by_day = df_resample_Y['counts'].sum()  # 統(tǒng)計(jì)每天的發(fā)帖量
count_by_day

post_time
2020-01-21     1
2020-01-22     1
2020-01-23     1
2020-01-24     0
2020-01-25     2
2020-01-26     0
2020-01-27     1
2020-01-28     2
2020-01-29     2
2020-01-30     3
2020-01-31     5
2020-02-01     8
2020-02-02     5
2020-02-03    11
2020-02-04    15
2020-02-05    13
2020-02-06     6
2020-02-07    14
2020-02-08    10
2020-02-09    10
2020-02-10    20
2020-02-11     9
2020-02-12    19
2020-02-13    15
2020-02-14     5
2020-02-15    12
2020-02-16    19
2020-02-17    45
2020-02-18    46
2020-02-19    43
2020-02-20    41
2020-02-21    49
2020-02-22    42
2020-02-23    33
2020-02-24    34
2020-02-25    29
2020-02-26    35
2020-02-27    36
2020-02-28    27
2020-02-29    27
2020-03-01    32
2020-03-02    26
2020-03-03    36
2020-03-04    40
2020-03-05    16
2020-03-06    22
2020-03-07    12
2020-03-08    11
2020-03-09    22
2020-03-10    19
2020-03-11    11
Freq: D, Name: counts, dtype: int64

from pandas.plotting import register_matplotlib_converters
register_matplotlib_converters()
plt.bar(count_by_day.index, count_by_day)  # 畫圖看一下論壇每天發(fā)帖數(shù)量趨勢

<BarContainer object of 51 artists>

mean_by_day = df_resample_Y['reply_num'].sum()  # 統(tǒng)計(jì)每天每貼的平均回復(fù)量
mean_by_day

post_time
2020-01-21     553.0
2020-01-22     317.0
2020-01-23     128.0
2020-01-24       0.0
2020-01-25     362.0
2020-01-26       0.0
2020-01-27     227.0
2020-01-28     271.0
2020-01-29      10.0
2020-01-30      46.0
2020-01-31     385.0
2020-02-01     820.0
2020-02-02      80.0
2020-02-03     530.0
2020-02-04     610.0
2020-02-05     574.0
2020-02-06    2097.0
2020-02-07     231.0
2020-02-08     470.0
2020-02-09     160.0
2020-02-10    2214.0
2020-02-11     552.0
2020-02-12     807.0
2020-02-13     502.0
2020-02-14      33.0
2020-02-15     386.0
2020-02-16     342.0
2020-02-17     264.0
2020-02-18    1046.0
2020-02-19     452.0
2020-02-20     772.0
2020-02-21    1374.0
2020-02-22     372.0
2020-02-23     730.0
2020-02-24     414.0
2020-02-25     274.0
2020-02-26     379.0
2020-02-27     673.0
2020-02-28     635.0
2020-02-29      66.0
2020-03-01     609.0
2020-03-02     117.0
2020-03-03     370.0
2020-03-04     256.0
2020-03-05     372.0
2020-03-06     106.0
2020-03-07      45.0
2020-03-08     454.0
2020-03-09     140.0
2020-03-10      57.0
2020-03-11      12.0
Freq: D, Name: reply_num, dtype: float64

plt.plot(mean_by_day.index, mean_by_day)  # 畫圖看一下論壇每天平均回復(fù)數(shù)的趨勢

[<matplotlib.lines.Line2D at 0x18948458358>]

從第2張圖看，論壇在二月初達(dá)到了討論高峰晴弃，群眾對(duì)于疫情發(fā)展即為關(guān)心掩幢。

詞頻統(tǒng)計(jì)

import jieba
import jieba.analyse
import wordcloud  # 詞云展示庫
from PIL import Image  # 圖像處理庫

news = df[['content', 'post_time']].sort_values('post_time')
with open('news_data.csv', 'w') as f:
    news.to_csv('news_data.csv')  # 先將數(shù)據(jù)存檔，避免后續(xù)調(diào)試常從頭開始上鞠。

news = pd.read_csv('news_data.csv')

news['content'].head()

0          丁香園上線「疫情地圖」际邻，幫你實(shí)時(shí)了解新型肺炎最新進(jìn)展！
1      最新芍阎！武漢同濟(jì)世曾、武漢協(xié)和同時(shí)發(fā)布新型冠狀病毒肺炎快速診療指南！
2                    新型肺炎 17 例死亡病例病情介紹
3            最權(quán)威的武漢肺炎流行病數(shù)據(jù)能曾，原來……（知乎咖喱雞）
4    不缺頂尖醫(yī)院度硝、有 SARS 前車之鑒，為何武漢仍然每一步都走晚了寿冕？
Name: content, dtype: object

punctuation = """【】★“”蕊程！，驼唱。藻茂？、~@#￥%……&*（）Ｃ悼摇辨赐？?――＂＃＄％＆＇<<>>（）＊＋－／：京办；＜＝＞＠［＼］＾＿｀｛｜｝～?????掀序、〃》「」『』【】〔〕〖〗?????〝〞????–—‘'?“”??…?﹏"""
re_punctuation = "[{}]+".format(punctuation)
remove_words = [u'的', u'，', u'和', u'是', u'隨著', u'對(duì)于', u'對(duì)', u'等', u'能', u'都', u'惭婿。', u' ', u'不恭、',
                u'中', u'-', u'在', u'了', u'通常', u'如果', u'我們', u'需要', u'什么', u'下', u'一', u'嗎', u'有']  # 自定義去除詞庫

news_list = []
for new in tqdm(news['content']):
    new = str(new)
    new = re.sub(re_punctuation, '', new)  # 去掉一些沒用的符號(hào)

    seg_list_exact = jieba.cut(new, cut_all=False)  # 精確模式分詞
    for word in seg_list_exact:
        if word not in remove_words:  # 如果不在去除詞庫中
            news_list.append(word)  # 將分詞加入 list 中叶雹。

news_list[:10]

['丁香', '園', '上線', '疫情', '地圖', '幫', '你', '實(shí)時(shí)', '了解', '新型']

詞頻統(tǒng)計(jì)

word_c = Counter(news_list)  # 對(duì)分詞詞頻統(tǒng)計(jì)
word_top10 = word_c.most_common(20)  # 獲取前20的詞
print(word_top10)

 [('肺炎', 297), ('新冠', 286), ('冠狀病毒', 132), ('疫情', 128), ('新型', 121), ('病毒', 86), ('感染', 65), ('治療', 61), ('患者', 59), ('武漢', 57), ('醫(yī)生', 48), ('2', 48), ('方案', 47), ('|', 47), ('醫(yī)院', 44), ('確診', 43), ('病例', 41), ('月', 41), ('診療', 36), ('關(guān)于', 34)]

wc = wordcloud.WordCloud(font_path='C:\Windows\Fonts\msyh.ttc', max_words=100, max_font_size=60)
wc.generate_from_frequencies(word_c)  # 從字典生成詞云
plt.imshow(wc)  # 顯示詞云
plt.axis('off')  # 關(guān)閉坐標(biāo)軸
plt.show()  # 顯示圖像

最后通過顯示詞云，可以直觀的體現(xiàn)論壇里人民對(duì)于疫情的討論熱點(diǎn)