1 Word2vec（gensim）

1.1 Word2vec介紹

word2vec是一個將單詞轉換成向量形式的工具嵌莉。可以把對文本內容的處理簡化為向量空間中的向量運算冲呢，計算出向量空間上的相似度棚点，來表示文本語義上的相似度冲甘。

1.2 gensim(word2vec)的安裝與使用

1.2.1 安裝gensim

安裝gensim工具包，有以下要求:

python>=2.6
NumPy>=1.3
Scipy>=0.7

打開Anaconda Prompt叁鉴，輸入

 pip install gensim

有以下內容土涝，安裝即為成功。

1.2.2 gensim word2vec的使用

gensim中word2vec介紹：

word2vec(sentences=None, size=100, alpha=0.025, window=5, min_count=5, max_vocab_size=None, sample=1e-3, seed=1, workers=3, min_alpha=0.0001,sg=0, hs=0, negative=5, cbow_mean=1, hashfxn=hash, iter=5, null_word=0,trim_rule=None, sorted_vocab=1, batch_words=MAX_WORDS_IN_BATCH, compute_loss=False)

word2vec的參數介紹：

sg defines the training algorithm. By default (sg=0), CBOW is used.Otherwise (sg=1), skip-gram is employed.
size is the dimensionality of the feature vectors.
window is the maximum distance between the current and predicted word within a sentence.
alpha is the initial learning rate (will linearly drop to min_alpha as training progresses).
seed = for the random number generator. Initial vectors for eachword are seeded with a hash of the concatenation of word + str(seed).Note that for a fully deterministically-reproducible run, you must also limit the model toa single worker thread, to eliminate ordering jitter from OS thread scheduling. (In Python3, reproducibility between interpreter launches also requires use of the PYTHONHASHSEEDenvironment variable to control hash randomization.)
min_count = ignore all words with total frequency lower than this.
max_vocab_size = limit RAM during vocabulary building; if there are more uniquewords than this, then prune the infrequent ones. Every 10 million word typesneed about 1GB of RAM. Set to
None for no limit (default).
sample = threshold for configuring which higher-frequency words are randomly downsampled; default is 1e-3, useful range is (0, 1e-5).
workers = use this many worker threads to train the model (=faster training with multicore machines).hs = if 1, hierarchical softmax will be used for model training.If set to 0 (default), and
negative is non-zero, negative sampling will be used.negative = if > 0, negative sampling will be used, the int for negativespecifies how many "noise words" should be drawn (usually between 5-20).Default is 5. If set to 0, no negative samping is used.
cbow_mean = if 0, use the sum of the context word vectors. If 1 (default), use the mean.Only applies when cbow is used.
hashfxn = hash function to use to randomly initialize weights, for increasedtraining reproducibility. Default is Python's rudimentary built in hash function.
iter = number of iterations (epochs) over the corpus. Default is 5.
trim_rule = vocabulary trimming rule, specifies whether certain words should remainin the vocabulary, be trimmed away, or handled using the default (discard if word count < min_count).Can be None (min_count will be used), or a callable that accepts parameters (word, count, min_count) andreturns either utils.RULE_DISCARD, utils.RULE_KEEP or utils.RULE_DEFAULT.Note: The rule, if given, is only used to prune vocabulary during build_vocab() and is not stored as partof the model.
sorted_vocab = if 1 (default), sort the vocabulary by descending frequency beforeassigning word indexes.
batch_words= target size (in words) for batches of examples passed to worker threads (andthus cython routines). Default is 10000. (Larger batches will be passed if individualtexts are longer than 10000 words, but the standard cython code truncates to that maximum.)

準備語料庫:

中文的或者英文的文章都可以幌墓，一般要經過預處理才能使用但壮，將文本語料進行分詞冀泻，以空格,tab隔開都可以。

導入包：

import gensim.models as g
from gensim.models.word2vec import LineSentence
'''Word2vec的輸入是一個LineSentence的迭代器蜡饵，即我們需要將原始的訓練語料轉化成一個sentence的迭代器弹渔；每一次迭代返回的sentence是一個word（utf8格式）的列表。我們再用這個迭代器作為輸入溯祸，構造一個Gensim內建的word2vec模型的對象肢专。
'''
# data/Corpus.txt為輸入的文件
model=g.Word2Vec(LineSentence('data/Corpus.txt'),size=100,window=1,min_count=1)

以上便完成了一個word2vec模型的訓練。你也可以根據需求修改其他的參數來訓練模型焦辅。

保存訓練結果：

# 將訓練的詞向量結果保存至data/vectors.bin文件博杖，一般將文件保存為二進制文件，方便以后做研究用筷登。
model.save('data/vectors.bin')
# 為了方便查看訓練的詞向量結果剃根，也可以將訓練的結果保存至data/vectors.txt文本文件。
model.wv.save_word2vec_format('data/vectors.txt', binary=False)

1.3 Word2vec使用舉例

1.3.1 訓練中文詞向量

中文語料庫：這里只是列舉了其開始的一小部分

經典教程轉載教程目錄簡介數據式數據準備關聯規(guī)則購物籃分析分類回歸聚類分析簡介

實驗代碼：

import gensim.models as g
from gensim.models.word2vec import LineSentence

model=g.Word2Vec(LineSentence('data/1.txt'),size=50,min_count=1)
model.save('data/v.bin')
model.wv.save_word2vec_format('data/v.txt', binary=False)

data/1.txt為輸入的語料庫前方，data/v.bin為訓練得到的二進制文件狈醉，data/v.txt為得到的詞向量的文本文件。得到的v.txt文件如下：下面只是截取該文件中的一小部分結果惠险。

 710 50
屬性 -0.009596 -0.001876 -0.009559 0.006456 -0.001698 0.003129 0.003461 -0.008876 0.007711 -0.007966 0.008706 0.008594 -0.000639 0.006059 -0.001408 0.004246 0.000866 0.005963 0.006523 -0.001072 -0.004322 -0.005270 -0.004433 -0.007570 0.006196 0.005732 0.003178 -0.001564 0.008695 -0.004273 -0.000454 0.006022 0.003671 -0.002460 -0.005034 -0.008246 0.008214 0.005232 0.008977 0.009046 -0.009300 0.003446 -0.003139 -0.008507 0.005131 -0.003137 0.001671 -0.000145 0.002956 0.008733
weka -0.001554 -0.002667 0.005671 -0.003087 0.005874 -0.000982 -0.007489 -0.003619 -0.001746 -0.002489 -0.007203 -0.006696 -0.004924 -0.005163 -0.004303 0.007519 -0.009520 0.000178 0.008966 0.003525 -0.003593 -0.009662 -0.001394 0.002259 -0.006288 -0.007043 0.002655 0.006285 -0.007610 -0.007114 -0.005075 0.007908 0.001376 0.006226 0.009289 0.004669 -0.002740 -0.005563 0.001656 -0.006386 0.001319 -0.005669 0.001278 0.001255 0.009341 0.005373 -0.005182 0.004410 0.005824 0.005403

查看‘經典’的詞向量：

s=model['經典']
print (s)

[-0.00151591  0.00092584 -0.009939   -0.00224788  0.00265429 -0.00093409  -0.00179082 -0.00541331  0.00329962 -0.00698855 -0.00517856 -0.00500181  0.00651171 -0.00661191  0.00882049  0.0098754   0.00071282 -0.00142486  0.00129473 -0.00415983  0.00480736 -0.00090799  0.00340422  0.00832723  -0.00304851  0.00366337 -0.00927676  0.0067507   0.00159891  0.00384319 -0.00919439 -0.00999665  0.00552959  0.00835639  0.00578091 -0.00271975  -0.00355495  0.00936656  0.00503161 -0.00182825  0.00873035  0.00328094 0.00860831 -0.00161888 -0.00698135 -0.00649323  0.00175485 -0.00052322 -0.00751577  0.00466034]

3.2 訓練英文詞向量

英文語料庫：這里只是列舉了其開始的一小部分

anarchism originated as a term of abuse first used against early working class radicals including the diggers of the english revolution and the sans culottes of the french revolution whilst the term is still used in a pejorative way to describe any act that used violent means to destroy the organization of society it has also been taken up as a positive label by self defined anarchists the word anarchism is derived from the greek without archons ruler chief king anarchism as a political philosophy is the belief that rulers are unnecessary and should be abolished although there are differing interpretations of what this means anarchism also refers to related social movements