kaggle賽題鏈接Home Depot Product Search Relevance，這個(gè)題目關(guān)鍵點(diǎn)就是特征提取布讹，給的數(shù)據(jù)需要觀察處理笆檀，提交的成績(jī)，排在前10%

第一類特征(詞匯語(yǔ)意)

• 可以用Levenshtein.ratio函數(shù)來(lái)評(píng)估兩個(gè)英文單詞相似度证舟，
• 使用nltk工具，nltk.corpus 中 wordnet來(lái)判斷兩個(gè)詞語(yǔ)意相似度,本文對(duì)原文進(jìn)行wordnet判斷窗骑，提取詞干后女责，詞發(fā)生很大變化，詞干提取后主要用來(lái)計(jì)算tidf,wordvec
• 如果以上兩個(gè)相似度都很低创译，還要查看屬性文件中是否有匹配單詞（只發(fā)現(xiàn)一個(gè)訓(xùn)練集是三分抵知，但是與title、description十分不匹配软族，但是與屬性文檔中一個(gè)項(xiàng)匹配）
• 如果以上都不匹配刷喜，至少發(fā)現(xiàn)四個(gè)案例是這樣，搜索的產(chǎn)品型號(hào)立砸，需要使用google搜索（網(wǎng)絡(luò)請(qǐng)求）掖疮，用搜索到的第一個(gè)內(nèi)容再來(lái)判斷相似度

第二類特征 詞向量（gensim中wod2vec）

• 用word2vec訓(xùn)練維基百科英文語(yǔ)料，來(lái)衡量?jī)蓚€(gè)詞匯相關(guān)性
• 用word2vec將product_title與product_description合起來(lái)作為語(yǔ)料訓(xùn)練得到詞向量

第三類特征 tidf

讀取數(shù)據(jù)

import pandas as pd

# 讀取數(shù)據(jù)
df_train = pd.read_csv('/Users/tangqinglong/Desktop/Scikit-learn/Depot/train.csv', encoding='ISO-8859-1')
df_test = pd.read_csv('/Users/tangqinglong/Desktop/Scikit-learn/Depot/test.csv', encoding='ISO-8859-1')
df_pro = pd.read_csv('/Users/tangqinglong/Desktop/Scikit-learn/Depot/product_descriptions.csv', encoding='ISO-8859-1')
df_attr = pd.read_csv('/Users/tangqinglong/Desktop/Scikit-learn/Depot/attributes.csv', encoding='ISO-8859-1')

設(shè)置pandas顯示颗祝，這樣可以全文顯示浊闪，方便查看

pd.set_option('display.max_colwidth',1000)

豎合恼布，將test數(shù)據(jù)追加到train下方，并且忽略index

df_all = pd.concat((df_train, df_test), axis=0,ignore_index=True)

將產(chǎn)品描述信息作為一列加入到總表中

df_all = pd.merge(df_all, df_pro, how='left', on='product_uid')

看一下打分基本分布情況

import copy
a = copy.deepcopy(df_train['relevance'].values)
a.sort()
import matplotlib.pyplot as plt
print type(a)
plt.plot(a)
plt.show()

df_y = pd.DataFrame(df_train['relevance'])
df_y['num'] = 1
df_y_g = df_y['num'].groupby(df_y['relevance'])
df_y_g.sum()

輸出

relevance
1.00     2105
1.25        4
1.33     3006
1.50        5
1.67     6780
1.75        9
2.00    11730
2.25       11
2.33    16060
2.50       19
2.67    15202
2.75       11
3.00    19125
Name: num, dtype: int64

可以看到評(píng)分從1分到3分搁宾，隔0.33或者0.34一個(gè)檔次折汞，其中1.25，1.50等等樣本太少盖腿，可以忽略不計(jì)爽待，分成7類，可以把回歸問(wèn)題轉(zhuǎn)為分類問(wèn)題來(lái)考慮奸忽，先按照回歸問(wèn)題處理

• 建立停用詞字典

dic_stopwords = dict(zip(stopwords.words('english'),xrange(len(stopwords.words('english')))))

• 文本預(yù)處理

from nltk import SnowballStemmer  
from nltk.corpus import stopwords
import re
import Levenshtein
stemmer = SnowballStemmer('english')

pattern_replace_pair_list = [
            (r"<.+?>", r""),
            # html codes
            (r"&nbsp;", r" "),
            (r"&amp;", r"&"),
            (r"&#39;", r"'"),
            (r"/>/Agt/>", r""),
            (r"</a<gt/", r""),
            (r"gt/>", r""),
            (r"/>", r""),
            (r"<br", r""),
            # do not remove [".", "/", "-", "%"] as they are useful in numbers, e.g., 1.97, 1-1/2, 10%, etc.
            (r"[ &<>)(_,;:!?\+^~@#\$\*]+", r" "),
            (r"'s\\b", r""),
            (r"[']+", r""),
    # 將DeckOver這樣次分開(kāi)堕伪，字母與字符英文連在一起的也分開(kāi)
            #(r'([A-Z][a-z]+|[a-z]+|\d+)', r'\1 '),
            (r'(\d?)([a-zA-Z]+)', r'\1 \2 '),
            #(r'(/d+)', r' \1 '),
            (r'([A-Z][a-z]+)', r' \1 '),
        ]
dic = {1:'one', 2:'two', 3:'three', 4:'four', 5:'five',6:'six', 7:'seven', 8:'eight', 9:'night',0:'zero'}
def dashrep(matchobj):
    if len(matchobj.group())==1:
        return dic[int(matchobj.group())]
    else:
        return matchobj.group() 

# 小寫(xiě) 去除標(biāo)點(diǎn)符號(hào)，停用詞
def transform(text):
    for pattern, replace in pattern_replace_pair_list:
        try:
            text = re.sub(pattern, replace, text)
        except:
            pass
    #text = re.sub(r'[\d]+', dashrep, text)
    text = re.sub(r"\s+", " ", text).strip()
    return ' '.join([word for word in text.lower().split() if word not in dic_stopwords])

#word_list = "Package stopwords is already up-to-date".split(" ")
#filtered_words = [word for word in word_list if word not in stopwords.words('english')]

# 詞干提取
def str_stemmer(s):
    # 不進(jìn)行詞干提取栗菜，只是變小寫(xiě)
    if isinstance(s, float):
        s = unicode(s)
    return ' '.join([stemmer.stem(word) for word in s.split()])

# str2 中有多少單詞在str1中
def str_common_word(str1, str2):
    return sum(int(str2.find(word)>=0) for word in str1.split())

def str_notcommon_word(str1, str2):
    return sum(int(str2.find(word)==-1) for word in str1.split())

# str1:title,str2:pro,str3:desc
# 功能：判斷標(biāo)題中有幾個(gè)單詞共同出現(xiàn)在str2,str3中
def str_common_desc_pro_word(str1, str2, str3):
    return sum(int(str2.find(word)>=0 and str3.find(word)>=0) for word in str1.split())

def word_vs_word_ratio(str1, str2):
    ratio = 0
    count = 0
    for word1 in str1.split():
        for word2 in str2.split():
            ratio = Levenshtein.ratio(word1, word2)+ratio
            count+=1
    return ratio/max(count,1)

def search_vs_word_ratio(str_search, str_des):
    ratio = 0
    if len(str_search) ==0:
        return 0
    for word in str_des:
        ratio = max(Levenshtein.ratio(str_search, word), ratio)
    return ratio



import nltk
import Levenshtein
def similarity(word1, word2):    
    from nltk.corpus import wordnet as wn
    word_1 = wn.synsets(word1)
    word_2 = wn.synsets(word2)
    sl = 0.
    for el1 in word_1:
        for el2 in word_2:
            val = el1.path_similarity(el2)
            if val is not None:
                sl = max(val,sl)
                if sl > 0.8:
                    break
    return sl

def similarity_sentences(str1, str2):
    sl, count, Ntotal, Nmatch = 0., 0., 0., 0.
    for word1 in nltk.pos_tag(str1.split()):
        score = 0
        for word2 in nltk.pos_tag(str2.split()):
            score = max(Levenshtein.ratio(word1[0],word2[0]),score)
            if score < 0.75:
                if word1[1][0]==word2[1][0]:
                    score = max(similarity(word1[0], word2[0]),score)
            #print score, word1, word2
            if score < 0.75:
                continue
            sl += score
            count += 1
            break
        if score > 0.7:
            Nmatch += 1
    if count == 0:
        return [0., 0.]
    return  [sl/count,Nmatch/max(len(str1.split()),1)] 

• 去除標(biāo)點(diǎn)符號(hào)，變小寫(xiě)蹄梢，去停用詞

%time df_all['search_term_transform'] = df_all['search_term'].map(lambda x:transform(x))
%time df_all['product_title_transform'] = df_all['product_title'].map(lambda x:transform(x))
%time df_all['pro_des_trans'] = df_all['product_description'].map(lambda x:transform(x))

• 詞干提取

%time df_all['search_term_transform_stem'] = df_all['search_term_transform'].map(lambda x:str_stemmer(x))
%time df_all['product_title_transform_stem'] = df_all['product_title_transform'].map(lambda x:str_stemmer(x))
%time df_all['pro_des_trans_stem'] = df_all['pro_des_trans'].map(lambda x:str_stemmer(x))

df_all['all_texts_transform'] = df_all['product_title_transform'] + ' . ' + df_all['pro_des_trans']
df_all['all_texts_trans_stemm'] = df_all['product_title_transform_stem'] + ' . ' + df_all['pro_des_trans_stem']

from joblib import Parallel, delayed
def func_similarity(df):
    sea_ter_tit = df.apply(lambda temp:similarity_sentences(temp['search_term_transform'],temp['product_title_transform']), axis=1).values[0]
    sea_ter_des = df.apply(lambda temp:similarity_sentences(temp['search_term_transform'],temp['pro_des_trans']), axis=1).values[0]
    sea_ter_all = df.apply(lambda temp:similarity_sentences(temp['search_term_transform'],temp['all_texts_transform']), axis=1).values[0]
    
    df.loc[:, 'sea_ter_tit'] = sea_ter_tit[0]
    df.loc[:, 'sea_ter_tit_ratio'] = sea_ter_tit[1]
    df.loc[:, 'sea_ter_des'] = sea_ter_des[0]
    df.loc[:, 'sea_ter_des_ratio'] = sea_ter_des[1]
    df.loc[:, 'sea_ter_all'] = sea_ter_all[0]
    df.loc[:, 'sea_ter_all_ratio'] = sea_ter_all[1]
    
    return df
def apply_parallel(df_grouped, func):
    """利用 Parallel 和 delayed 函數(shù)實(shí)現(xiàn)并行運(yùn)算"""
    results = Parallel(n_jobs=-1)(delayed(func)(group) for name, group in df_grouped)
    return pd.concat(results)

df_model = pd.DataFrame({})

使用多進(jìn)程計(jì)算特征

df_grouped =df_all.groupby(df_all.index)
%time df_model = apply_parallel(df_grouped, func_similarity)[['sea_ter_tit', \
                                                              'sea_ter_tit_ratio',\
                                                              'sea_ter_des',\
                                                              'sea_ter_des_ratio',\
                                                              'sea_ter_all',\
                                                              'sea_ter_all_ratio',\
                                                              'product_uid']]

import numpy as np
df_model['len_search_term']=df_all['search_term_transform_stem'].map(
        lambda x:len(x.split())).astype(np.int64)
#【新特征1】: search_term 和 product_title比較
df_model['dist_in_title'] = df_all.apply(lambda x:word_vs_word_ratio((x['search_term_transform_stem']),x['product_title_transform_stem']), axis=1)

df_model['dist_in_title1'] = df_all.apply(lambda x:search_vs_word_ratio((x['search_term_transform_stem']),x['product_title_transform_stem']), axis=1)

#【新特征2】: search_term 和 product_description比較
df_model['dist_in_desc'] = df_all.apply(lambda x:word_vs_word_ratio((x['search_term_transform_stem']),x['pro_des_trans_stem']), axis=1)

df_model['dist_in_desc1'] = df_all.apply(lambda x:search_vs_word_ratio((x['search_term_transform_stem']),x['pro_des_trans_stem']), axis=1)

df_model['len_of_query']=df_all['search_term_transform_stem'].map(
        lambda x:len(x.split())).astype(np.int64)
df_model['len_search'] = df_all['search_term_transform_stem'].map(lambda x:len(x))

# 產(chǎn)品標(biāo)題中有多少關(guān)鍵詞重合
df_model['commons_in_title']=df_all.apply(
    lambda x:str_common_word(
    x['search_term_transform_stem'],x['product_title_transform_stem']),axis=1)

# 描述中有多少關(guān)鍵詞重合
%time df_model['commons_in_desc'] = df_all.apply(lambda x:str_common_word(x['search_term_transform_stem'], x['pro_des_trans_stem']), axis=1)

%time df_model['common_in_desc_pro']=df_all.apply(lambda x: str_common_desc_pro_word(x['search_term_transform_stem'],x['pro_des_trans_stem'],x['product_title_transform_stem']), axis=1)

df_model['nn_word_in_search'] = df_all['search_term_transform_stem'].map(nn_word_numbers_In_Search)

df_model['queryvstitle'] = df_model.apply(lambda x: float(x['commons_in_title'])/max((x['len_of_query']),1), axis=1)

df_model['product_uid'] =df_all['product_uid']

df_model['queryvsdesc'] = df_model.apply(lambda x: float(x['commons_in_desc'])/max(x['len_of_query'],1), axis=1)

tidf特征

# 有了組合好的句子疙筹，可以分詞了準(zhǔn)備
# 分詞：這里我們用gensim，為了更加細(xì)致的分解TFIDF的步驟動(dòng)作禁炒；其實(shí)sklearn本身也有簡(jiǎn)單好用的tfidf模型
# Tokenize可以用各家或者各種方法而咆，就是把長(zhǎng)長(zhǎng)的string變成list of tokens。包括NLTK幕袱，SKLEARN都有自家的解決方案

from gensim.utils import tokenize
from gensim.corpora.dictionary import Dictionary
#得到了一個(gè)很多單詞的大詞典
dictionary = Dictionary(list(tokenize(x, errors='ignore')) for x in df_all['all_texts_trans_stemm'].values)
print(dictionary)
#這個(gè)類所做的事情也很簡(jiǎn)單暴备，就是掃便我們所有的語(yǔ)料，并且轉(zhuǎn)化成簡(jiǎn)單的單詞的個(gè)數(shù)計(jì)算
class MyCorpus(object):
    def __iter__(self):
        for x in df_all['all_texts_trans_stemm'].values:
            yield dictionary.doc2bow(list(tokenize(x, errors='ignore')))

# 這里這么折騰一下们豌，僅僅是為了內(nèi)存friendly涯捻。面對(duì)大量corpus數(shù)據(jù)時(shí)，你直接存成一個(gè)list望迎，會(huì)使得整個(gè)運(yùn)行變得很慢障癌。
# 所以我們搞成這樣，一次只輸出一組辩尊。但本質(zhì)上依舊長(zhǎng)得跟 [['sentence', '1'], ['sentence', '2'], ...]一樣
corpus = MyCorpus()

# 有了我們標(biāo)準(zhǔn)形式的語(yǔ)料庫(kù)涛浙，我們于是就可以init我們的TFIDFmodel了。這里做的事情摄欲，就是把已經(jīng)變成BoW向量的數(shù)組轿亮，做一次TFIDF的計(jì)算。
from gensim.models.tfidfmodel import TfidfModel
tfidf = TfidfModel(corpus)
#示例:這下我們看看一個(gè)普通的句子放過(guò)來(lái)長(zhǎng)什么樣子：
tfidf[dictionary.doc2bow(list(tokenize('hello world, good morning', errors='ignore')))]

#怎么判斷兩個(gè)句子的相似度呢胸墙？
#這里有個(gè)trick我注，因?yàn)槲覀兊玫降膖fidf只是『有這個(gè)字，就有這個(gè)值』劳秋，并不是一個(gè)全部值仓手。
#也就是說(shuō)胖齐，兩個(gè)matrix可能size是完全不一樣的。
#想用cosine計(jì)算的同學(xué)就會(huì)問(wèn)了嗽冒，兩個(gè)matrix的size都不fix呀伙，怎么辦？
#咦添坊，這里就注意咯剿另。他們的size其實(shí)是一樣的。只是把全部是0的那部分給省略了對(duì)吧贬蛙？
#于是雨女，我們只要拿其中一個(gè)作為index。擴(kuò)展開(kāi)全部的matrixsize阳准，另一個(gè)帶入氛堕，就可以計(jì)算了

from gensim.similarities import MatrixSimilarity

# 先把剛剛那句話包裝成一個(gè)方法
def to_tfidf(text):
    res = tfidf[dictionary.doc2bow(list(tokenize(text, errors='ignore')))]
    return res

# 然后，我們創(chuàng)造一個(gè)cosine similarity的比較方法
def cos_sim(text1, text2):
    tfidf1 = to_tfidf(text1)
    tfidf2 = to_tfidf(text2)
    index = MatrixSimilarity([tfidf1],num_features=len(dictionary))
    sim = index[tfidf2]
    # 本來(lái)sim輸出是一個(gè)array野蝇，我們不需要一個(gè)array來(lái)表示讼稚，
    # 所以我們直接cast成一個(gè)float
    return float(sim[0])

#計(jì)算搜索詞語(yǔ)與產(chǎn)品title相似度
df_model['tfidf_cos_sim_in_title'] = df_all.apply(lambda x: cos_sim(x['search_term_transform_stem'], x['product_title_transform_stem']), axis=1)
#計(jì)算搜索詞與產(chǎn)品描述description相似度
df_model['tfidf_cos_sim_in_desc'] = df_all.apply(lambda x: cos_sim(x['search_term_transform_stem'], x['pro_des_trans_stem']), axis=1)

通過(guò)Word2Vec來(lái)評(píng)判距離,搜索詞與產(chǎn)品title，產(chǎn)品描述的

import nltk
#1）nltk也是自帶一個(gè)強(qiáng)大的句子分割器绕沈∪裣耄【調(diào)用工具】
tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
#2）我們先把長(zhǎng)文本搞成list of 句子，再把句子變成list of 單詞：【文本->句子】
sentences = [tokenizer.tokenize(x) for x in df_all['all_texts_trans_stemm'].values]
#3）我們把list of lists 給 flatten了乍狐≡。【句子 -> 扁平化flatten】
sentences = [y for x in sentences for y in x] #一共1998321個(gè)句子。
#4）我們把句子里的單詞給分好浅蚪∨褐模可以用剛剛Gensim的tokenizer， 也可以用nltk的word_tokenizer 【句子 -> 單詞】
from nltk.tokenize import word_tokenize
w2v_corpus = [word_tokenize(x) for x in sentences]
#5） 訓(xùn)練我們的預(yù)料庫(kù)掘鄙，成為詞向量 【單詞 -> 訓(xùn)練語(yǔ)料庫(kù)model】
from gensim.models.word2vec import Word2Vec
model = Word2Vec(w2v_corpus, size=128, window=5, min_count=5, workers=4)

• 可以得到每個(gè)單詞的向量耘戚，但是每一格句子中由多個(gè)單詞組成，把每個(gè)單詞向量取平均,

import numpy as np
#6) 可以得到每個(gè)單詞的向量操漠，但是每一格句子中由多個(gè)單詞組成收津，把每個(gè)單詞向量取平均,
vocab = model.vocabulary
#得到任意text句子的vector（就是取平均）
def get_vector(text):
    res = np.zeros([128])
    count = 0
    for word in word_tokenize(text):
        res += model[word]
        count+=1
    return res/max(count,1)

• 計(jì)算兩個(gè)句子的vector的相似度, 用cosine similarity,用scipy的spatial功能

from scipy import spatial
def w2v_cos_sim(text1, text2):
    try:
        w2v1 = get_vector(text1)
        w2v2 = get_vector(text2)
        sim = 1 - spatial.distance.cosine(w2v1, w2v2)
        return float(sim)
    except:
        return float(0)

df_model['w2v_cos_sim_in_title'] = df_all.apply(lambda x: w2v_cos_sim(x['search_term_transform_stem'], x['product_title_transform_stem']), axis=1)
df_model['w2v_cos_sim_in_desc'] = df_all.apply(lambda x: w2v_cos_sim(x['search_term_transform_stem'], x['pro_des_trans_stem']), axis=1)

處理異常數(shù)據(jù)

def check_series(series):
    i = 0
    for el in series:
        if np.isnan(el):
            series[i] = 0
        i+=1
    return series
df_model.apply(check_series)

# 記錄測(cè)試集的id
test_ids = df_test['id']
# 分離出y_train
y_train = df_train['relevance'].values
X_train = df_model.loc[df_train.index]
X_test = df_model.loc[len(df_train.index):]

from sklearn import preprocessing
scaler = preprocessing.StandardScaler().fit(X_train.values)
train = scaler.transform(X_train.values)
test = scaler.transform(X_test.values)

相關(guān)性系數(shù)

from scipy.stats import pearsonr

lable=y_train
lr = []
for i, line in enumerate(X_train.values.T):
    lr.append([pearsonr(lable,line),i])
lr.sort()
print lr

xgboost

• 由于xgboost不支持logcosh,需要自定義損失函數(shù)

import xgboost as xgb
import random
import numpy as np

def huber_approx_obj(preds, dtrain):
    d = dtrain.get_label() - preds  #remove .get_labels() for sklearn
    h = 1  #h is delta in the graphic
    scale = 1 + (d / h) ** 2
    scale_sqrt = np.sqrt(scale)
    grad = d / scale_sqrt
    hess = 1 / scale / scale_sqrt
    return grad, hess

def log_cosh_obj(preds, dtrain):
    x = dtrain.get_label() - preds
    grad = np.tanh(-x)
    hess = 1- np.tanh(x)**2
    return grad, hess
def square_loss(preds, dtrain):
    #x = dtrain.get_label()-preds
    grad = preds - dtrain.get_label()
    hess = [1]*len(grad)
    return grad,hess
xgb_params = { 
'eta': 0.03, 
'max_depth': 6, 
'gamma':2,  # 在樹(shù)的葉子節(jié)點(diǎn)下一個(gè)分區(qū)的最小損失，越大算法模型越保守 浊伙。[0:]
'subsample': 0.6, 
'colsample_bytree': 0.7, 
'objective': 'reg:linear', 
'eval_metric': 'rmse', 
'silent': 1 ,
'min_child_weight':1,
'lambda':100,
'seed':1,
'booster':'gbtree'
#'booster':'gblinear'
}

# xgboost加載數(shù)據(jù)為DMatrix對(duì)象 
dtrain = xgb.DMatrix(train, y_train) 

# xgboost交叉驗(yàn)證并輸出rmse 
cv_output = xgb.cv(xgb_params, dtrain, num_boost_round=3000, early_stopping_rounds=100,obj = log_cosh_obj,
verbose_eval=50,nfold=5, show_stdv=False, shuffle=True) 
cv_output[['train-rmse-mean', 'test-rmse-mean']].plot() 

bst = xgb.train(xgb_params, dtrain, num_boost_round=3000, early_stopping_rounds=None, obj=log_cosh_obj)

dtest = xgb.DMatrix(test)
xgb_preds = bst.predict(dtest)
xgb_preds = scale_pred(xgb_preds)

把預(yù)測(cè)值控制到1到3

def scale_pred(pred):
    for i, el in enumerate(pred):
        if el > 3:
            pred[i] = 3
        elif el < 1:
            pred[i] = 1
    return pred

pd.DataFrame({'id':test_ids, 'relevance':xgb_preds[:,0]}).to_csv(
    '/Users/tangqinglong/Desktop/Scikit-learn/Depot/submission.csv', index=False)

keras+tf

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.optimizers import SGD

# Generate dummy data
import numpy as np

model = Sequential()
# Dense(64) is a fully-connected layer with 64 hidden units.
# in the first layer, you must specify the expected input data shape:
# here, 20-dimensional vectors.
L=len(X_train.columns)
model.add(Dense(64, activation='relu', input_dim=L))
model.add(Dropout(0.5))
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1))

sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='logcosh',
              optimizer=sgd,
              metrics=['mse'])

model.fit(train, y_train,
          epochs=1000,
          batch_size=256)

模型融合撞秋，把不同模型預(yù)測(cè)的成績(jī)作為特征，再進(jìn)行回歸