大師兄的數(shù)據(jù)分析學(xué)習(xí)筆記(三十一）：機器學(xué)習(xí)模型總結(jié)
大師兄的數(shù)據(jù)分析學(xué)習(xí)筆記(三十三）：模型評估（二）

一、分類模型評估

1. 二分類

• 二分類就是標(biāo)注分類時有兩類的分類瘤旨，在數(shù)據(jù)挖掘中是常見的類型语御。
• 通常會將二分類中更被關(guān)注的類定義成正類，用數(shù)字1來表示梨睁。
• 另一個類定義成負(fù)類济瓢，用數(shù)字0表示帚戳。

• 有時0和1不是直接得到的，而是經(jīng)過模型輸出后被劃分為正類的概率：

• 這時需要確定一個閾值（比如0.5）通铲，大于閾值為1毕莱，否則為0。

2. 混淆矩陣

• 如果把測試集的真實分類和經(jīng)過模型預(yù)測后最終的判別結(jié)果進(jìn)行整理颅夺，會得到四種映射關(guān)系：

• 如果把上圖中各類映射的數(shù)量數(shù)出來朋截，并整理成為一個矩陣的形式，就是混淆矩陣吧黄。

• 矩陣中的每一行代表一個真實的分類
• 矩陣中的每一列代表一個預(yù)測的分類
• 如果把四個映射找到混淆矩陣中的位置部服，則如圖：

• 可以看出，對角線上的分類屬于正確分類拗慨。
• 不在對角線上的分類數(shù)據(jù)錯誤分類廓八。
• 所以理想的模型應(yīng)該是一個對角陣，如果得不到對角陣赵抢，對角線上的數(shù)字加和占統(tǒng)治地位也是可以的剧蹂。

3. 關(guān)鍵指標(biāo)

• 可以通過混淆矩陣獲得關(guān)鍵指標(biāo)。
• 正確率(Accuracy Rate)：
• 召回率(Recall,True Positive Rate)：
• F-measure(正確率和召回率的權(quán)衡值)：
• 準(zhǔn)確率(Precision)：
• 錯誤接收率(FPR)：
• 錯誤拒絕率(FRR)：

4. 多元混淆矩陣

• 與二分類不同烦却，多分類中的每個類都是被關(guān)注的宠叼。
• 多分類也可以制作成混淆矩陣，同樣對角線上的值表示正確值短绸。

• 準(zhǔn)確率：和二分類保持一致车吹。
• 召回率/F-measure：

1. 先計算所有的TP筹裕、FN等醋闭，再以二分類方法計算。
2. 分別把每個類當(dāng)做正類朝卒，都算一個召回率/F-measure证逻，然后取加權(quán)或者不加權(quán)的平均值。

• 如果值是模型輸出后被劃分的概率抗斤，可以使用ROC和AUC：

4.1 ROC

• ROC(Receiver Operating characteristic Curve)可以很容易查出任意界限值時的對性能的識別能力 囚企。
  

• 首先將模型輸出的預(yù)測結(jié)果得分從大到小進(jìn)行排列：

  
  
• 將不同閾值得出的關(guān)鍵指標(biāo)結(jié)果畫到坐標(biāo)系上丈咐，連成一條線：
  

4.2 AUC

• AUC（Area Under Curve）被定義為ROC下與坐標(biāo)軸圍成的面積。
• 由于ROC一般都處于y=x直線的上方龙宏，所以AUC的取值范圍在0.5和1之間棵逊。
• AUC越接近1.0，檢測方法真實性越高;等于0.5時银酗，則真實性最低辆影，無應(yīng)用價值。

4.3 增益圖

• 增益圖可以在宏觀上反應(yīng)分類器的分類效果黍特。
  

4.4 KS圖

• KS圖可以通過TPR和FPR的差距蛙讥，反映出對正類樣本分類的區(qū)分度。
  

5. 代碼實現(xiàn)

>>>import os
>>>import numpy as np
>>>import pandas as pd
>>>import tensorflow as tf
>>>import matplotlib.pyplot as plt
>>>from sklearn.model_selection import train_test_split
>>>from sklearn.metrics import roc_curve,auc,roc_auc_score
>>>from sklearn.preprocessing import StandardScaler
>>>from keras.models import Sequential
>>>from keras.layers.core import Dense,Activation

>>>df = pd.read_csv(os.path.join(".", "data", "WA_Fn-UseC_-HR-Employee-Attrition.csv"))

>>>X_tt,X_validation,Y_tt,Y_validation = train_test_split(df.JobLevel,df.JobSatisfaction,test_size=0.2)
>>>StandardScaler().fit_transform(np.array(X_tt).reshape(-1,1))
>>>X_train,X_test,Y_train,Y_test = train_test_split(X_tt,Y_tt,test_size=0.25)

>>>mdl = Sequential()
>>>mdl.add(Dense(50))
>>>mdl.add(Activation("sigmoid"))
>>>mdl.add(Dense(2))
>>>mdl.add(Activation("softmax"))
>>>mdl.compile(loss="mean_squared_error",optimizer=tf.keras.optimizers.SGD(lr=0.05))
>>>mdl.fit(X_train,np.array([[0,1] if i==1 else [1,0] for i in Y_train]),epochs=50,batch_size=800)

>>>f = plt.figure()

>>>xy_lst = [(X_train,Y_train),(X_validation,Y_validation),(X_test,Y_test)]
>>>for i in range(len(xy_lst)):
>>>    X_part = xy_lst[i][0]
>>>    Y_part = [0 if x<=1 else 1 for x in xy_lst[i][1]]

>>>    Y_pred = mdl.predict(X_part)
>>>    Y_pred = np.array(Y_pred[:,1]).reshape((1,-1))[0]
>>>    f.add_subplot(1,3,i+1)
>>>    fpr,tpr,threshold = roc_curve(Y_part,Y_pred)
>>>    plt.plot(fpr,tpr)
>>>    print("NN","AUC",auc(fpr,tpr))
>>>    print("NN","AUC_Score",roc_auc_score(Y_part,Y_pred))
>>>    print("="*40)
>>>plt.show()
Epoch 1/50
2/2 [==============================] - 0s 1ms/step - loss: 0.2929
Epoch 2/50
2/2 [==============================] - 0s 1ms/step - loss: 0.2168
Epoch 3/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1858
Epoch 4/50
2/2 [==============================] - 0s 1ms/step - loss: 0.1738
Epoch 5/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1673
Epoch 6/50
2/2 [==============================] - 0s 0s/step - loss: 0.1638
Epoch 7/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1623
Epoch 8/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1616
Epoch 9/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1613
Epoch 10/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1606
Epoch 11/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1602
Epoch 12/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1599
Epoch 13/50
2/2 [==============================] - 0s 1ms/step - loss: 0.1598
Epoch 14/50
2/2 [==============================] - 0s 0s/step - loss: 0.1596
Epoch 15/50
2/2 [==============================] - 0s 1ms/step - loss: 0.1596
Epoch 16/50
2/2 [==============================] - 0s 0s/step - loss: 0.1596
Epoch 17/50
2/2 [==============================] - 0s 1ms/step - loss: 0.1596
Epoch 18/50
2/2 [==============================] - 0s 0s/step - loss: 0.1596
Epoch 19/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 20/50
2/2 [==============================] - 0s 0s/step - loss: 0.1596
Epoch 21/50
2/2 [==============================] - 0s 0s/step - loss: 0.1596
Epoch 22/50
2/2 [==============================] - 0s 1ms/step - loss: 0.1595
Epoch 23/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 24/50
2/2 [==============================] - 0s 0s/step - loss: 0.1596
Epoch 25/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 26/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1595
Epoch 27/50
2/2 [==============================] - 0s 0s/step - loss: 0.1595
Epoch 28/50
2/2 [==============================] - 0s 1ms/step - loss: 0.1595
Epoch 29/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1595
Epoch 30/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 31/50
2/2 [==============================] - 0s 0s/step - loss: 0.1596
Epoch 32/50
2/2 [==============================] - 0s 0s/step - loss: 0.1596
Epoch 33/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 34/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1595
Epoch 35/50
2/2 [==============================] - 0s 0s/step - loss: 0.1595
Epoch 36/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1595
Epoch 37/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 38/50
2/2 [==============================] - 0s 0s/step - loss: 0.1596
Epoch 39/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 40/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1595
Epoch 41/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 42/50
2/2 [==============================] - 0s 1ms/step - loss: 0.1595
Epoch 43/50
2/2 [==============================] - 0s 1ms/step - loss: 0.1596
Epoch 44/50
2/2 [==============================] - 0s 0s/step - loss: 0.1597
Epoch 45/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 46/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 47/50
2/2 [==============================] - 0s 1ms/step - loss: 0.1596
Epoch 48/50
2/2 [==============================] - 0s 0s/step - loss: 0.1595
Epoch 49/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1596
Epoch 50/50
2/2 [==============================] - 0s 1000us/step - loss: 0.1595
28/28 [==============================] - 0s 444us/step
NN AUC 0.4800573010558846
NN AUC_Score 0.4800573010558846
========================================
10/10 [==============================] - 0s 556us/step
NN AUC 0.5361630625365283
NN AUC_Score 0.5361630625365283
========================================
10/10 [==============================] - 0s 667us/step
NN AUC 0.5459870673259795
NN AUC_Score 0.5459870673259795
========================================