一莺债、文本預(yù)處理階段###
1.1 設(shè)定訓(xùn)練集和測(cè)試集
訓(xùn)練集每一類(lèi)的數(shù)量為500個(gè)文檔，測(cè)試集每一類(lèi)的數(shù)量也為500個(gè)文檔。
1.2 計(jì)算每個(gè)文本的DF
為每一個(gè)文本計(jì)算TF，return格式為：'word', 'file_name', term-frequency
先算出每個(gè)文檔中的'word', term-frequency, 在結(jié)束改文本的循環(huán)后將該文本中出現(xiàn)的詞以 'word', 'file_name', term-frequency的形式加入 word_docid_tf
def compute_tf_by_file(self):
word_docid_tf = []
for name in self.filenames:
with open(join(name), 'r') as f:
tf_dict = dict()
for line in f:
line = self.process_line(line)
words = jieba.cut(line.strip(), cut_all=False)
for word in words:
tf_dict[word] = tf_dict.get(word, 0) + 1
tf_list = tf_dict.items()
word_docid_tf += [[item[0], name, item[1]] for item in tf_list]
return word_docid_tf
1.3 計(jì)算每個(gè)詞項(xiàng)的TF粪小、DF
為每一個(gè)詞項(xiàng)計(jì)算TF，return的term_freq格式為：'word', dict ( 'file_name ', tf )
為每一個(gè)詞項(xiàng)計(jì)算DF抡句，return的doc_freq格式為：'word', df
def compute_tfidf(self):
word_docid_tf = self.compute_tf_by_file()
word_docid_tf.sort()
doc_freq = dict()
term_freq = dict()
for current_word, group in groupby(word_docid_tf, itemgetter(0)):
doclist = []
df = 0
for current_word, file_name, tf in group:
doclist.append((file_name, tf))
df += 1
term_freq[current_word] = dict(doclist)
doc_freq[current_word] = df
return term_freq, doc_freq
1.4 精簡(jiǎn)term_freq, doc_freq
除去只出現(xiàn)在一個(gè)或0個(gè)文檔中的詞項(xiàng)
除去數(shù)字詞項(xiàng)
def reduce_tfidf(self, term_freq, doc_freq):
remove_list = []
for key in term_freq.keys():
if len(key) < 2:#該詞只出現(xiàn)在一個(gè)或0個(gè)文檔中
remove_list.append(key)
else:
try:
float(key)#該詞是數(shù)字
remove_list.append(key)
except ValueError:
continue
for key in remove_list:
term_freq.pop(key)
doc_freq.pop(key)
return term_freq, doc_freq
1.5 為每個(gè)文本構(gòu)建特征向量train_feature, train_target
為term_freq, doc_freq中的key探膊，也就是詞項(xiàng)標(biāo)明index
用jieba分詞，將分好的詞放入一個(gè)臨時(shí)的數(shù)組中待榔。
遍歷數(shù)組逞壁，由doc_freq[word]取得DF并計(jì)算iDF流济，由term_freq[word][name]
取得該詞項(xiàng)在該文檔中的TF，并計(jì)算每個(gè)詞項(xiàng)的tf-idf值腌闯，并作為向量中詞項(xiàng)對(duì)應(yīng)index那一維的值绳瘟。
train_feature, train_target = train_tfidf.tfidf_feature(os.path.join(input_path, 'train'),train_tf, train_df, N)
def tfidf_feature(self, dir, term_freq, doc_freq, N):
filenames = []
for (dirname, dirs, files) in os.walk(dir):
for file in files:
filenames.append(os.path.join(dirname, file))
word_list = dict()
for idx, word in enumerate(doc_freq.keys()):
word_list[word] = idx
features = []
target = []
for name in filenames:
feature = np.zeros(len(doc_freq.keys()))
words_in_this_file = set()
tags = re.split('[/\\\\]', name)
tag = tags[-2]
with open(name, 'rb') as f:
for line in f:
line = self.process_line(line)
words = jieba.cut(line.strip(), cut_all=False)
for word in words:
words_in_this_file.add(word)
for word in words_in_this_file:
try:
idf = np.log(float(N) / doc_freq[word])
tf = term_freq[word][name]
feature[word_list[word]] = tf*idf
except KeyError:
continue
features.append(feature)
target.append(tag)
return sparse.csr_matrix(np.asarray(features)), np.asarray(target)
1.6 存儲(chǔ)&加載
為了節(jié)約之后運(yùn)行的時(shí)間，可以通過(guò)如下方式把測(cè)試集tf和df的值直接存儲(chǔ)：
Pickle.dump(train_tf, open(os.path.join(input_path, 'train_tf.pkl'), 'wb'))
print "saved train_tf.pkl"
Pickle.dump(train_df, open(os.path.join(input_path, 'train_df.pkl'), 'wb'))
print "saved train_df.pkl"
之后運(yùn)行時(shí)姿骏，可以通過(guò)如下方式把測(cè)試集tf和df的值直接加載到內(nèi)存糖声，省去了重新計(jì)算的時(shí)間：
train_tf = Pickle.load(open(os.path.join(input_path, 'train_tf.pkl'), 'rb'))
print "loaded train_tf.pkl"
train_df = Pickle.load(open(os.path.join(input_path, 'train_df.pkl'), 'rb'))
train_tfidf.doc_freq=train_df
print "loaded train_df.pkl"
