一般來(lái)說(shuō)學(xué)習(xí)曲線：一種用來(lái)判斷訓(xùn)練模型的一種方法通過(guò)查看學(xué)習(xí)曲線，可以對(duì)模型的狀態(tài)進(jìn)行判斷慕的。

1.偏差的方差：

: 偏差度量了學(xué)習(xí)算法的期望預(yù)測(cè)與真實(shí)結(jié)果的偏離程序, 即。
: 方差度量了同樣大小的訓(xùn)練集的變動(dòng)所導(dǎo)致的學(xué)習(xí)性能的變化, 即 。

請(qǐng)看下圖：

分類器偏差方差關(guān)系

2.learning curve

如何判斷模型是否合理?

判斷模型擬合程度

這里可以使用learning curve幫我們判斷

上圖分別對(duì)應(yīng)了欠擬合芹缔，過(guò)擬合和合理學(xué)習(xí)曲線坯癣，這里需要解釋一下，橫坐標(biāo)表示訓(xùn)練集大小最欠，縱坐標(biāo)表示metric score示罗。芝硬。如下圖所示:

學(xué)習(xí)曲線訓(xùn)練方法

在時(shí)蚜点，訓(xùn)練集和驗(yàn)證集的cv結(jié)果隨著訓(xùn)練集數(shù)據(jù)增大逐漸收斂，但是正確率遠(yuǎn)遠(yuǎn)于我們期望的情況拌阴，這時(shí)屬于绍绘，可以通過(guò): 訓(xùn)練更長(zhǎng)時(shí)間, 增加樣本, 增加特征、增加樹(shù)的深度皮官、減小正則項(xiàng)脯倒，增加網(wǎng)絡(luò)結(jié)構(gòu)，如增加隱藏層數(shù)目捺氢， 尋找合適的網(wǎng)絡(luò)架構(gòu)藻丢，使用更大的NN結(jié)構(gòu) 來(lái)解決

在時(shí)，訓(xùn)練集上面的cv結(jié)果非常高摄乒，但是驗(yàn)證集的結(jié)果就不是那么理想悠反，這時(shí)屬于，可以使用更多的數(shù)據(jù)馍佑，正則化(regularization)斋否， 尋找合適的網(wǎng)絡(luò)結(jié)構(gòu)，樹(shù)的剪枝來(lái)解決拭荤。

一般來(lái)說(shuō)茵臭，過(guò)擬合的情況出現(xiàn)的會(huì)比較多一些，除非在計(jì)算資源很缺乏舅世，一般不會(huì)出現(xiàn)欠擬合的情況

這里可以使用sklearn learning curve這個(gè)函數(shù), 學(xué)習(xí)曲線請(qǐng)看下圖：

學(xué)習(xí)曲線

# -*- coding: utf-8 -*-
"""
-------------------------------------------------
   File Name：     aaa
   Description :
   Author :        Asdil
   date：          2020/3/26
-------------------------------------------------
   Change Activity:
                   2020/3/26:
-------------------------------------------------
"""
__author__ = 'Asdil'
import plotly
import plotly.io as pio
import plotly.offline as py  # 離線模式
from plotly.graph_objs import Data, Figure
plotly.offline.init_notebook_mode(connected=True)
import warnings
warnings.filterwarnings('ignore')

def add_trace(percent, score_means, score_std, is_train=True):
    """add_trace方法用于添加每輪驗(yàn)證結(jié)果

    Parameters
    ----------
    percent : list
        訓(xùn)練數(shù)據(jù)集百分比 eg: [0, 0.25, 0.5, 0.75, 1]
    score_means : list
        訓(xùn)練百分n的數(shù)據(jù)后, cv得分均值
    score_std : list
        訓(xùn)練百分n的數(shù)據(jù)后, cv得分均方差
    is_train : bool
        True 或者 False
    Returns
    ----------
    """
    score_means = np.array(score_means)
    score_std = np.array(score_std)

    name = 'Train score' if is_train else 'Test score'
    color_std = "rgba(211, 47, 47, 0.4)" if is_train else "rgba(56, 142, 60, 0.4)"
    color_mean = "#d32f2f" if is_train else "#388E3C"
    trace_mean = {
        "line": {"color": color_mean},
        "name": name,
        "type": "scatter",
        "x": percent,
        "y": score_means,
        "showlegend": True
    }
    trace_mean_add_std = {
        "fill": "tonexty",
        "line": {
            "color": color_std,
            "width": 0.1
        },
        "mode": "lines",
        "name": "",
        "type": "scatter",
        "x": percent,
        "y": score_means+score_std,
        "hoverlabel": {"namelength": 20},
        "showlegend": False
    }
    trace_mean_sub_std = {
        "line": {
            "color": color_std,
            "width": 0.1
        },
        "mode": "lines",
        "name": "",
        "type": "scatter",
        "x": percent,
        "y": score_means-score_std,
        "hoverlabel": {"namelength": 20},
        "showlegend": False
    }

    return [trace_mean_sub_std, trace_mean_add_std, trace_mean]


def create_lay_out(score_name, title='Learning curve'):
    """create_lay_out方法用于添加layout注釋

    Parameters
    ----------
    title : str
        標(biāo)題
    score_name : str
        metric 函數(shù)名稱 F1, ACC ect...
    Returns
    ----------
    """
    layout = {
        "title": {"text": f"{title}"},
        "xaxis": {"title": {"text": "訓(xùn)練集樣本百分比"}},
        "yaxis": {"title": {"text": f"{score_name}"}},
        "legend": {
            "x": 0.8,
            "y": 0
        },
        "autosize": True
    }
    return layout

from sklearn.naive_bayes import GaussianNB
import numpy as np
from sklearn.svm import SVC
from sklearn.datasets import load_digits
from sklearn.model_selection import learning_curve
from sklearn.model_selection import ShuffleSplit

# 這里使用ShuffleSplit 對(duì)訓(xùn)練集合劃分
cv = ShuffleSplit(n_splits=100, test_size=0.2, random_state=0)
X, y = load_digits(return_X_y=True) # 使用手寫體數(shù)據(jù)集minst
estimator = SVC(gamma=0.001) # smv模型

percent = np.linspace(.1, 1.0, 5) # 注意這里將訓(xùn)練集按照10%-100%劃分成5份，對(duì)應(yīng)結(jié)果圖片x坐標(biāo)有5個(gè)
scoring = 'f1_macro' # f1_macro accuracy 
train_sizes, train_scores, test_scores, fit_times, _ = learning_curve(estimator, 
                                                                      X, 
                                                                      y, 
                                                                      cv=cv, 
                                                                      n_jobs=-1,
                                                                      train_sizes=percent,
                                                                      scoring = scoring,
                                                                      return_times=True)

# 獲取5次訓(xùn)練集f1得分均值和方差查辩，驗(yàn)證集5次f1均值和方差
train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
# 這是訓(xùn)練集分批cv結(jié)果的那條線
train_trace = add_trace(percent=percent, 
                        score_means=train_scores_mean, 
                        score_std=train_scores_std, is_train=True)
# 這是測(cè)試集集cv結(jié)果的那條線
test_trace = add_trace(percent=percent, 
                        score_means=test_scores_mean, 
                        score_std=test_scores_std, is_train=False)

layout = create_lay_out(score_name=scoring, title='Learning curve of SVC')
data = Data([*train_trace, *test_trace])
fig = Figure(data=data, layout=layout)
py.iplot(fig)
# pio.write_html(fig, file='iris.html')

學(xué)習(xí)曲線動(dòng)態(tài)圖

3.損失函數(shù)值可視化：

上面的例子小樣本集中是可以嘗試的胖笛，但是通常情況下，不會(huì)有那么多的自己算資源宜岛，假設(shè)一個(gè)模型需要運(yùn)行2天长踊，我們將訓(xùn)練集拆分成不同百分比驗(yàn)證模型顯然是不可取的，因?yàn)闀r(shí)間消耗太大谬返。更多的時(shí)候我們需要看訓(xùn)練集cv和驗(yàn)證集cv的曲線變化之斯，這里介紹另一個(gè)可以實(shí)時(shí)查看損失和metrc的包，名叫lrcurve遣铝，需要注意的是這個(gè)包需要安裝tensorlow
這里使用一個(gè)pytorch的例子佑刷，這里使用一個(gè)手寫體識(shí)別的例子，在例子中我們會(huì)顯示出訓(xùn)練集和測(cè)試集的損失函數(shù)和metric

from lrcurve import PlotLearningCurve
import torch
from torchvision import datasets, transforms
import torch.nn as nn
from torch import optim
import torch.nn.functional as F
import torchvision.transforms as transforms
from sklearn.metrics import accuracy_score, f1_score
import numpy as np

transform = transforms.Compose([transforms.ToTensor(),
                              transforms.Normalize((0.5,), (0.5,)),
                              ])
# 下載數(shù)據(jù)集可能需要花一點(diǎn)時(shí)間
trainset = datasets.MNIST('./mnist', download=True, train=True, transform=transform)
valset = datasets.MNIST('./mnist', download=True, train=False, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
valloader = torch.utils.data.DataLoader(valset, batch_size=64, shuffle=True)
dataiter = iter(trainloader)
images, labels = dataiter.next()
input_size = 784
hidden_sizes = [64, 32]
output_size = 10
# 定義一個(gè)簡(jiǎn)單的全連接模型
model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.LogSoftmax(dim=1))
criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.003, momentum=0.9)
epochs = 15 # 進(jìn)行15輪

plot = PlotLearningCurve(
   # 這里定義三幅圖酿炸，分別是損失函數(shù)圖, accuracy曲線和f1 socre曲線
    facet_config = {
        'loss': { 'name': 'NLLLoss', 'limit': [0, None] },
        'accuracy': { 'name': 'Accuracy', 'limit': [0, 1] }, # acc 和 f1 取值都是0-1
        'f1_score': { 'name': 'F1_score', 'limit': [0, 1] }

    },
    xaxis_config = { 'name': 'Epoch', 'limit': [0, 15] } # 這里定義x軸瘫絮，我們迭代epochs = 15輪，因此這里是0-15
)
# 將整個(gè)pytorch訓(xùn)練過(guò)程包在plot中
with plot:
    for epoch in range(epochs):
        train_loss = 0
        train_metric_1 = 0
        train_metric_2 = 0
        for images, labels in trainloader:
            images = images.view(images.shape[0], -1)
            optimizer.zero_grad()
            output = model(images)
            loss = criterion(output, labels)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()
           #記錄每個(gè)epoch中每個(gè)batch的acc和f1
            train_metric_1 += f1_score(np.argmax(output.detach().numpy(), axis=1), labels.detach().numpy(), 
                                       average='macro')
            train_metric_2 += accuracy_score(np.argmax(output.detach().numpy(), axis=1), labels.detach().numpy())
        model.eval() # eval mode
        val_loss = 0
        val_metric_1 = 0
        val_metric_2 = 0
        with torch.no_grad():
            for images, labels in valloader:
                images = images.view(images.shape[0], -1)
                output = model(images)
                loss = criterion(output, labels)
                val_loss += loss.item()
                 #記錄每個(gè)epoch中每個(gè)batch的acc和f1
                val_metric_1 += f1_score(np.argmax(output.detach().numpy(), axis=1), labels.detach().numpy(), 
                                         average='macro')
                val_metric_2 += accuracy_score(np.argmax(output.detach().numpy(), axis=1), labels.detach().numpy())
        # 求出所有batch平均值
        train_metric_1 = train_metric_1/len(trainloader)  
        train_metric_2 = train_metric_2/len(trainloader)   
        val_metric_1 = val_metric_1/len(valloader)
        val_metric_2 = val_metric_2/len(valloader)
        # 將結(jié)果添加到學(xué)習(xí)曲線中以顯示
        plot.append(epoch, {
            'loss': {
                'train': train_loss/len(trainloader), # 每個(gè)epoch所有batch損失函數(shù)均值
                'validation': val_loss/len(valloader),
            },
            'accuracy': {
                'train': train_metric_2, # 訓(xùn)練集acc均值
                'validation': val_metric_2
            },
            'f1_score': {
                'train': train_metric_1,  # 訓(xùn)練集f1數(shù)均值
                'validation': val_metric_1
            },
        })
        plot.draw() # 每一個(gè)epoch將數(shù)據(jù)顯示出來(lái)

學(xué)習(xí)曲線

這實(shí)際上是一個(gè)動(dòng)圖填硕，可以參考下面這張圖

學(xué)習(xí)曲線動(dòng)圖演示

4.使用tensorboard

還有一種方法可視化損失函數(shù)和metric那就是使用tensorboard麦萤，方法也很簡(jiǎn)單,還是上面那個(gè)例子,前面的定義都一樣：

# 應(yīng)用tensorboard
from torch.utils.tensorboard import SummaryWriter
# default `log_dir` is "runs" - we'll be more specific here
writer = SummaryWriter('runs/fashion_mnist_experiment_1') # 日志路徑

for epoch in range(epochs):
    train_loss = 0
    train_metric_1 = 0
    train_metric_2 = 0
    for images, labels in trainloader:
        images = images.view(images.shape[0], -1)
        optimizer.zero_grad()
        output = model(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()
        train_metric_1 += f1_score(np.argmax(output.detach().numpy(), axis=1), labels.detach().numpy(), 
                                       average='macro')
        train_metric_2 += accuracy_score(np.argmax(output.detach().numpy(), axis=1), labels.detach().numpy())
    model.eval() # eval mode
    val_loss = 0
    val_metric_1 = 0
    val_metric_2 = 0
    with torch.no_grad():
        for images, labels in valloader:
            images = images.view(images.shape[0], -1)
            output = model(images)
            loss = criterion(output, labels)
            val_loss += loss.item()
            val_metric_1 += f1_score(np.argmax(output.detach().numpy(), axis=1), labels.detach().numpy(), 
                                         average='macro')
            val_metric_2 += accuracy_score(np.argmax(output.detach().numpy(), axis=1), labels.detach().numpy())

    train_metric_1 = train_metric_1/len(trainloader)  
    train_metric_2 = train_metric_2/len(trainloader)   
    val_metric_1 = val_metric_1/len(valloader)
    val_metric_2 = val_metric_2/len(valloader)
    # 這里把信息寫入tensorboard
    writer.add_scalars('loss',
                       {'train loss':train_loss/len(trainloader), 
                        'val_loss:':val_loss/len(valloader)},epoch)
        
writer.close()

# 這里直接在jupyter中顯示
%load_ext tensorboard
%tensorboard --logdir='runs/fashion_mnist_experiment_1'

tensorboard顯示學(xué)習(xí)曲線