最近爬取了一些國(guó)外的圖片，由于標(biāo)簽都是英文，對(duì)文本檢索不友好，所以就搞了這個(gè)單詞翻譯工具纯露；很多推薦算法在計(jì)算物品與物品相似距離時(shí)經(jīng)常用到同現(xiàn)矩陣，那么舉一反三 該算法是否同樣可以應(yīng)用在求解中英單詞相似距離上呢代芜？
算法背景
同現(xiàn)矩陣計(jì)算公式
- 通過案例說明
N(i)與N(j)分別表示喜歡 i 物品的人數(shù)與喜歡 j 物品的人數(shù)埠褪，上述公式大致意思是求解喜歡物品 i 的人中又同時(shí)喜歡物品 j 的占比是多少，比值越大越能說明兩個(gè)物品的關(guān)聯(lián)度高挤庇，那么當(dāng)其它用戶去購買物品 i 時(shí)將在很大程度上喜歡物品 j 钞速；不過需要注意的時(shí)如果物品 j 是一個(gè)熱門物品，那么很多人都會(huì)喜歡物品 j嫡秕，極端情況下所有喜歡物品 i 的用戶都喜歡物品 j , 那么計(jì)算出的物品 i 與物品 j 就是高度相似的渴语，為了避免熱門物品的影響，在分母上對(duì)熱門物品進(jìn)行了懲罰昆咽，當(dāng)N(j)很大時(shí)驾凶，相識(shí)度就會(huì)很低牙甫。
另外該方式還有另一個(gè)優(yōu)勢(shì)，那就是在計(jì)算相似度時(shí)不需要額外收集評(píng)分?jǐn)?shù)據(jù)
基于python實(shí)現(xiàn)
- 數(shù)據(jù)處理
import re
# import jieba
import unicodedata
from LAC import LAC
lac = LAC(mode='seg')
def unicode_to_ascii(s):
return ''.join(c for c in unicodedata.normalize('NFD', s)
if unicodedata.category(c) != 'Mn')
def preprocess_eng(w):
w = unicode_to_ascii(w.lower().strip())
# creating a space between a word and the punctuation following it
# eg: "he is a boy." => "he is a boy ."
# Reference:- https://stackoverflow.com/questions/3645931/
# python-padding-punctuation-with-white-spaces-keeping-punctuation
# w = re.sub(r"([?.!,])", r" \1 ", w)
w = re.sub(r"([?.!,])", r" ", w)
# replace several spaces with one space
w = re.sub(r'[" "]+', " ", w)
# replacing everything with space except (a-z, A-Z, ".", "?", "!", ",")
w = re.sub(r"[^a-zA-Z?.!,]+", " ", w)
w = w.rstrip().strip()
# adding a start and an end token to the sentence
# so that the model know when to start and stop predicting.
# w = '<start> ' + w + ' <end>'
return w.split(' ')
def preprocess_chinese(w):
w = unicode_to_ascii(w.lower().strip())
w = re.sub(r'[" "]+', "", w)
w = w.rstrip().strip()
# w = " ".join(list(w)) # add the space between words
# w = '<start> ' + w + ' <end>'
return list(lac.run(w))
input_texts = open('D:/Download/英中機(jī)器文本翻譯/ai_challenger_translation_train_20170904/translation_train_data_20170904/train.en', 'r', encoding='UTF-8').read().splitlines()
target_texts = open('D:/Download/英中機(jī)器文本翻譯/ai_challenger_translation_train_20170904/translation_train_data_20170904/train.zh', 'r', encoding='UTF-8').read().splitlines()
input_texts_d = []
target_texts_d = []
i = 1
for it, tt in zip(input_texts, target_texts):
input_texts_d.append(preprocess_eng(it))
target_texts_d.append(preprocess_chinese(tt))
- 計(jì)算相似性
mydic = {}
kvdic = {}
for it_ks, tt_ks in zip(input_texts_d, target_texts_d):
for en_k in it_ks:
en_v = {}
if en_k in mydic:
en_v = mydic.get(en_k)
for zh_k in tt_ks:
if zh_k in en_v:
en_v[zh_k] += 1
else:
en_v[zh_k] = 1
mydic[en_k] = en_v
if en_k in kvdic:
kvdic[en_k] += 1
else:
kvdic[en_k] = 1
for zh_k in tt_ks:
if zh_k in kvdic:
kvdic[zh_k] += 1
else:
kvdic[zh_k] = 1
res = {}
for k, v in mydic.items():
zh_dic = {}
for zh, cn in v.items():
zh_dic[zh] = round(cn/(kvdic[zh]*kvdic[k])**0.5, 5)
res[k] = sorted(zh_dic.items(), key= lambda kv: kv[1], reverse=True)[:5]
效果展示
寫在最后
現(xiàn)實(shí)生活中對(duì)一件物品或商品進(jìn)行描述時(shí)往往會(huì)有很多詞匯和短語调违，如 “紅薯” 和 “地瓜” 窟哺；大家可以嘗試使用上面算法來挖掘內(nèi)容中的同義詞...
