目的
給定一個(gè)或多個(gè)搜索詞或详，如“高血壓 患者”狭园，從已有的若干篇文本中找出最相關(guān)的(n篇)文本沙咏。
理論知識(shí)
文本檢索（text retrieve）的常用策略是：用一個(gè)ranking function根據(jù)搜索詞對(duì)所有文本進(jìn)行排序宴胧，選取前n個(gè)晶姊，就像百度搜索一樣柬讨。
算法：模型選擇
- 1崩瓤、基于word2vec的詞語(yǔ)相似度計(jì)算模型
- 2、python的實(shí)現(xiàn)用到了gensim庫(kù)
- 3踩官、“jieba”中文分詞
分步實(shí)現(xiàn)：
- jieba.cut
方法接受三個(gè)輸入?yún)?shù): 需要分詞的字符串却桶；cut_all 參數(shù)用來(lái)控制是否采用全模式；HMM 參數(shù)用來(lái)控制是否使用
HMM 模型
構(gòu)建停用詞表
"""分詞蔗牡、去停用詞"""
# stopwords =["項(xiàng)目",'招標(biāo)','中標(biāo)', '公告', '標(biāo)題'] # 停用詞
stopwords = []
stopwords = [ w.strip() for w in stopwords ]
結(jié)巴分詞后的停用詞性 [標(biāo)點(diǎn)符號(hào)颖系、連詞、助詞辩越、副詞嘁扼、介詞、時(shí)語(yǔ)素黔攒、‘的’偷拔、數(shù)詞、方位詞亏钩、代詞]
# stop_flag = ['x', 'c', 'u', 'd', 'p', 't', 'uj', 'm', 'f', 'r'] # 停用詞性
stop_flag = []
對(duì)一篇文章分詞莲绰、去停用詞
def tokenization(filename):
result = []
with open(filename, 'r') as f:
text = f.read()
words = pseg.cut(text)
for word, flag in words:
if flag not in stop_flag and word not in stopwords:
result.append(word)
return result
對(duì)目錄下的所有文本進(jìn)行預(yù)處理，構(gòu)建字典
corpus = [];
dirname = 'demo/articles'
filenames = []
for f in files:
corpus.append(tokenization(text))
filenames.append(text)
dictionary = corpora.Dictionary(corpus)
print len(dictionary)
建立詞袋模型
# 生成詞向量
doc_vectors = [dictionary.doc2bow(text) for text in corpus] # 語(yǔ)料庫(kù)
建立TF-IDF模型
# 生成TF-IDF模型
tfidf = models.TfidfModel(doc_vectors) # TF-IDF模型對(duì)語(yǔ)料庫(kù)建模
tfidf_vectors = tfidf[doc_test_vec] # 每個(gè)詞的TF-IDF值
相似矩陣計(jì)算相似度
index = similarities.MatrixSimilarity(tfidf[doc_vectors])
sim = index[tfidf[doc_test_vec]] # 獲取分值索引
print(sim)
相似度排序
scores=sorted(enumerate(sim), key=lambda item: -item[1]) # 排序
print(scores[0][1]) # 輸出分值
結(jié)果示例：
測(cè)試數(shù)據(jù)為： 富寧縣里達(dá)中學(xué)宿舍樓建設(shè)項(xiàng)目
匹配結(jié)果集（匹配度從大到泄贸蟆） [(2, 1.0), (31, 0.07981655), (43, 0.077732354), (33, 0.06620947), (30, 0.065360494), (14, 0.061563488), (6, 0.05077639), (22, 0.05062699), (7, 0.044322222), (42, 0.044024862), (21, 0.043359876), (26, 0.035853535), (27, 0.03457492), (29, 0.033902794), (45, 0.03236963), (25, 0.031936638), (40, 0.030814772), (48, 0.030788476), (20, 0.027607089), (8, 0.02558621), (11, 0.024541285), (5, 0.024447413), (28, 0.020779021), (4, 0.020459857), (13, 0.015429099), (34, 0.014453442), (50, 0.011855431), (36, 0.006562164), (0, 0.006476198), (32, 0.0051991176), (46, 0.00477116), (35, 0.0047449875), (38, 0.004728446), (18, 0.004499278), (41, 0.004158474), (44, 0.0037516006), (47, 0.0036311403), (15, 0.003384664), (37, 0.00318741), (23, 0.0030692797), (17, 0.0022487652), (39, 0.0020392523), (24, 0.0016430109), (12, 0.0014699087), (1, 0.0), (3, 0.0), (9, 0.0), (10, 0.0), (16, 0.0), (19, 0.0), (49, 0.0)]
分析結(jié)果為:中標(biāo)項(xiàng)目：富寧縣里達(dá)中學(xué)宿舍樓建設(shè)項(xiàng)目 最大匹配度為 1.0
測(cè)試數(shù)據(jù)為： 濕地保護(hù)與恢復(fù)建設(shè)工程
匹配結(jié)果集（匹配度從大到懈蚯） [(13, 0.57420367), (40, 0.10633894), (48, 0.106248185), (43, 0.10532686), (49, 0.0816016), (12, 0.077999234), (31, 0.07725123), (25, 0.07712983), (11, 0.058984473), (50, 0.05736675), (7, 0.047928438), (34, 0.04754001), (33, 0.04504219), (30, 0.038571842), (22, 0.037484765), (27, 0.03233484), (45, 0.031974725), (14, 0.0313408), (26, 0.030683806), (5, 0.030661184), (2, 0.026870431), (4, 0.02638424), (8, 0.026375605), (20, 0.02581845), (35, 0.024404963), (32, 0.019936334), (28, 0.019432766), (44, 0.018292043), (42, 0.018038727), (38, 0.01745583), (6, 0.017230202), (17, 0.015729848), (46, 0.013131632), (29, 0.012461022), (19, 0.0117950225), (47, 0.0064870343), (0, 0.0), (1, 0.0), (3, 0.0), (9, 0.0), (10, 0.0), (15, 0.0), (16, 0.0), (18, 0.0), (21, 0.0), (23, 0.0), (24, 0.0), (36, 0.0), (37, 0.0), (39, 0.0), (41, 0.0)]
分析結(jié)果為:中標(biāo)項(xiàng)目：四川省若爾蓋國(guó)際重要濕地保護(hù)與恢復(fù)建設(shè)工程第1標(biāo)段 最大匹配度為 0.57420367
