R機(jī)器學(xué)習(xí)mlr3：模型評價(jià)和比較

前面一篇介紹了如何使用mlr3創(chuàng)建任務(wù)和學(xué)習(xí)器、擬合模型澎灸、預(yù)測和簡單的評價(jià)蛹头，本篇將模型評價(jià)的一些細(xì)節(jié)問題顿肺，展示mlr3如何使得這些步驟變得更加簡單！

二分類變量和ROC曲線

對于二分類變量渣蜗，結(jié)果有陰性和陽性兩種屠尊，而且判定陰性和陽性的閾值是可以認(rèn)為設(shè)定的。ROC曲線可以很好的幫助我們確定最佳的分割點(diǎn)耕拷。

首先看一下如何獲取一個(gè)分類變量的混淆矩陣：

library(mlr3verse)
## 載入需要的程輯包：mlr3
data("Sonar", package = "mlbench")
task <- as_task_classif(Sonar, target = "Class", positive = "M") # 指定陽性

learner <- lrn("classif.rpart", predict_type = "prob") # 指定預(yù)測類型
prediction <- learner$train(task)$predict(task)
conf <- prediction$confusion
print(conf)
##         truth
## response  M  R
##        M 95 10
##        R 16 87

繪制ROC曲線也是非常方便：

autoplot(prediction, type = "roc")

也可以非常方便的繪制PRC曲線：

autoplot(prediction, type = "prc")

重抽樣

mlr3支持的重抽樣方法：

cross validation ("cv"),
leave-one-out cross validation ("loo"),
repeated cross validation ("repeated_cv"),
otstrapping ("bootstrap"),
subsampling ("subsampling"),
holdout ("holdout"),
in-sample resampling ("insample"),
custom resampling ("custom").

查看重抽樣的方法：

library(mlr3verse)
as.data.table(mlr_resamplings)
##            key        params iters
## 1:   bootstrap ratio,repeats    30
## 2:      custom                  NA
## 3:   custom_cv                  NA
## 4:          cv         folds    10
## 5:     holdout         ratio     1
## 6:    insample                   1
## 7:         loo                  NA
## 8: repeated_cv folds,repeats   100
## 9: subsampling ratio,repeats    30

還有一些特殊類型的重抽樣方法可以通過擴(kuò)展包實(shí)現(xiàn)讼昆，比如mlr3spatiotemporal包。

默認(rèn)的方法是holdout：

resampling <- rsmp("holdout")
print(resampling)
## <ResamplingHoldout> with 1 iterations
## * Instantiated: FALSE
## * Parameters: ratio=0.6667

可以通過以下方法改變比例：

resampling$param_set$values <- list(ratio = 0.8)

# 或者
rsmp("holdout", ratio = 0.8)
## <ResamplingHoldout> with 1 iterations
## * Instantiated: FALSE
## * Parameters: ratio=0.8

下面一個(gè)例子使用5折交叉驗(yàn)證方法骚烧，建立一個(gè)決策樹模型：

library(mlr3verse)
task <- tsk("penguins") # 創(chuàng)建任務(wù)
learner <- lrn("classif.rpart", predict_type = "prob") # 創(chuàng)建學(xué)習(xí)器浸赫，設(shè)定預(yù)測的結(jié)果是概率
resampling <- rsmp("cv", folds = 5) # 選擇重抽樣方法

rr <- resample(task, learner, resampling, store_models = T) # 1行代碼搞定
## INFO  [20:47:12.966] [mlr3] Applying learner 'classif.rpart' on task 'penguins' (iter 5/5) 
## INFO  [20:47:12.996] [mlr3] Applying learner 'classif.rpart' on task 'penguins' (iter 1/5) 
## INFO  [20:47:13.010] [mlr3] Applying learner 'classif.rpart' on task 'penguins' (iter 2/5) 
## INFO  [20:47:13.019] [mlr3] Applying learner 'classif.rpart' on task 'penguins' (iter 4/5) 
## INFO  [20:47:13.029] [mlr3] Applying learner 'classif.rpart' on task 'penguins' (iter 3/5)
print(rr)
## <ResampleResult> of 5 iterations
## * Task: penguins
## * Learner: classif.rpart
## * Warnings: 0 in 0 iterations
## * Errors: 0 in 0 iterations

獲得平均的模型表現(xiàn)

rr$aggregate(msr("classif.acc"))
## classif.acc 
##   0.9448423

獲得單個(gè)模型的表現(xiàn)

rr$score(msr("classif.acc"))[,7:9]
##    iteration              prediction classif.acc
## 1:         1 <PredictionClassif[20]>   0.9710145
## 2:         2 <PredictionClassif[20]>   0.8985507
## 3:         3 <PredictionClassif[20]>   0.9130435
## 4:         4 <PredictionClassif[20]>   0.9710145
## 5:         5 <PredictionClassif[20]>   0.9705882

檢查警告或者錯(cuò)誤：

rr$warnings
## Empty data.table (0 rows and 2 cols): iteration,msg
rr$errors
## Empty data.table (0 rows and 2 cols): iteration,msg

取出單個(gè)模型

rr$learners[[5]]$model
## n= 276 
## 
## node), split, n, loss, yval, (yprob)
##       * denotes terminal node
## 
## 1) root 276 158 Adelie (0.427536232 0.206521739 0.365942029)  
##   2) flipper_length< 206.5 170  54 Adelie (0.682352941 0.311764706 0.005882353)  
##     4) bill_length< 43.35 117   4 Adelie (0.965811966 0.034188034 0.000000000) *
##     5) bill_length>=43.35 53   4 Chinstrap (0.056603774 0.924528302 0.018867925) *
##   3) flipper_length>=206.5 106   6 Gentoo (0.018867925 0.037735849 0.943396226)  
##     6) bill_depth>=17.2 8   4 Chinstrap (0.250000000 0.500000000 0.250000000) *
##     7) bill_depth< 17.2 98   0 Gentoo (0.000000000 0.000000000 1.000000000) *

這個(gè)包也可以和其他決策樹可視化R包無縫銜接，比如非常畫圖非常好看的rpart.plot:

library(rpart.plot)
## 載入需要的程輯包：rpart
rpart.plot(rr$learners[[5]]$model)

查看預(yù)測結(jié)果：

rr$prediction()
## <PredictionClassif> for 344 observations:
##     row_ids     truth  response prob.Adelie prob.Chinstrap prob.Gentoo
##           1    Adelie    Adelie  0.96969697     0.03030303  0.00000000
##           4    Adelie    Adelie  0.96969697     0.03030303  0.00000000
##          26    Adelie    Adelie  0.96969697     0.03030303  0.00000000
## ---                                                                   
##         333 Chinstrap Chinstrap  0.05660377     0.92452830  0.01886792
##         334 Chinstrap Chinstrap  0.05660377     0.92452830  0.01886792
##         335 Chinstrap Chinstrap  0.05660377     0.92452830  0.01886792

# 查看單個(gè)預(yù)測結(jié)果
rr$predictions()[[1]]
## <PredictionClassif> for 69 observations:
##     row_ids     truth  response prob.Adelie prob.Chinstrap prob.Gentoo
##           1    Adelie    Adelie  0.96969697     0.03030303  0.00000000
##           4    Adelie    Adelie  0.96969697     0.03030303  0.00000000
##          26    Adelie    Adelie  0.96969697     0.03030303  0.00000000
## ---                                                                   
##         338 Chinstrap Chinstrap  0.08888889     0.88888889  0.02222222
##         342 Chinstrap Chinstrap  0.08888889     0.88888889  0.02222222
##         344 Chinstrap Chinstrap  0.08888889     0.88888889  0.02222222

提取特定iteration的結(jié)果

rr$filter(c(3,5))
print(rr)
## <ResampleResult> of 2 iterations
## * Task: penguins
## * Learner: classif.rpart
## * Warnings: 0 in 0 iterations
## * Errors: 0 in 0 iterations

可視化結(jié)果：

task <- tsk("pima") # 非常著名的糖尿病數(shù)據(jù)集
task$select(c("glucose","mass"))
learner <- lrn("classif.rpart", predict_type = "prob")
resampling <- rsmp("cv")
rr <- resample(task, learner, resampling, store_models = T)
## INFO  [20:47:13.436] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 5/10) 
## INFO  [20:47:13.449] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 6/10) 
## INFO  [20:47:13.461] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 9/10) 
## INFO  [20:47:13.473] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 8/10) 
## INFO  [20:47:13.488] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/10) 
## INFO  [20:47:13.501] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/10) 
## INFO  [20:47:13.513] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 10/10) 
## INFO  [20:47:13.524] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/10) 
## INFO  [20:47:13.536] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 7/10) 
## INFO  [20:47:13.548] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/10)

autoplot(rr, measure = msr("classif.auc"))

ROC曲線：10折交叉驗(yàn)證平均后的：

autoplot(rr, type = "roc")

樹狀圖：

autoplot(rr, type = "prediction")

可視化單個(gè)模型：

rr1 <- rr$filter(1)

autoplot(rr1, type = "prediction")

所有支持的可視化類型可在此處找到：autoplot.ResampleResult

benchmark

用于比較多個(gè)模型止潘，比如多個(gè)模型在單個(gè)任務(wù)的表現(xiàn)掺炭、多個(gè)模型在多個(gè)任務(wù)的表現(xiàn)等，使用不同的預(yù)處理進(jìn)行的多個(gè)模型的表現(xiàn)等凭戴！

首先創(chuàng)建一個(gè)design

mlr3通過design進(jìn)行比較多個(gè)模型，這個(gè)design是包含Task炕矮、Learner么夫、Resampling的組合。

library(mlr3verse)

# 使用benchmark_grid函數(shù)創(chuàng)建
design <- benchmark_grid(
  tasks = tsks(c("spam", "german_credit", "sonar")),
  learners = lrns(c("classif.ranger", "classif.rpart", "classif.featureless"), predict_type = "prob"),
  resamplings = rsmps(c("holdout", "cv"))
)
print(design)
##                  task                         learner              resampling
##  1: <TaskClassif[49]>      <LearnerClassifRanger[37]> <ResamplingHoldout[19]>
##  2: <TaskClassif[49]>      <LearnerClassifRanger[37]>      <ResamplingCV[19]>
##  3: <TaskClassif[49]>       <LearnerClassifRpart[37]> <ResamplingHoldout[19]>
##  4: <TaskClassif[49]>       <LearnerClassifRpart[37]>      <ResamplingCV[19]>
##  5: <TaskClassif[49]> <LearnerClassifFeatureless[37]> <ResamplingHoldout[19]>
##  6: <TaskClassif[49]> <LearnerClassifFeatureless[37]>      <ResamplingCV[19]>
##  7: <TaskClassif[49]>      <LearnerClassifRanger[37]> <ResamplingHoldout[19]>
##  8: <TaskClassif[49]>      <LearnerClassifRanger[37]>      <ResamplingCV[19]>
##  9: <TaskClassif[49]>       <LearnerClassifRpart[37]> <ResamplingHoldout[19]>
## 10: <TaskClassif[49]>       <LearnerClassifRpart[37]>      <ResamplingCV[19]>
## 11: <TaskClassif[49]> <LearnerClassifFeatureless[37]> <ResamplingHoldout[19]>
## 12: <TaskClassif[49]> <LearnerClassifFeatureless[37]>      <ResamplingCV[19]>
## 13: <TaskClassif[49]>      <LearnerClassifRanger[37]> <ResamplingHoldout[19]>
## 14: <TaskClassif[49]>      <LearnerClassifRanger[37]>      <ResamplingCV[19]>
## 15: <TaskClassif[49]>       <LearnerClassifRpart[37]> <ResamplingHoldout[19]>
## 16: <TaskClassif[49]>       <LearnerClassifRpart[37]>      <ResamplingCV[19]>
## 17: <TaskClassif[49]> <LearnerClassifFeatureless[37]> <ResamplingHoldout[19]>
## 18: <TaskClassif[49]> <LearnerClassifFeatureless[37]>      <ResamplingCV[19]>

然后進(jìn)行比較肤视，也是1行代碼即可档痪！

bmr <- benchmark(design, store_models = T)
## INFO  [20:47:16.049] [mlr3] Running benchmark with 99 resampling iterations 
## INFO  [20:47:16.053] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 1/10) 
## INFO  [20:47:16.070] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 10/10) 
## INFO  [20:47:16.280] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 3/10) 
## INFO  [20:47:16.290] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 6/10) 
## INFO  [20:47:16.300] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 9/10) 
## INFO  [20:47:16.309] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 2/10) 
## INFO  [20:47:16.506] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 8/10) 
## INFO  [20:47:18.070] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 8/10) 
## INFO  [20:47:18.149] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 1/10) 
## INFO  [20:47:18.159] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 7/10) 
## INFO  [20:47:18.176] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 3/10) 
## INFO  [20:47:18.193] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 1/1) 
## INFO  [20:47:18.203] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 8/10) 
## INFO  [20:47:18.400] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 4/10) 
## INFO  [20:47:18.410] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 4/10) 
## INFO  [20:47:18.486] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 5/10) 
## INFO  [20:47:19.873] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 6/10) 
## INFO  [20:47:19.950] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 5/10) 
## INFO  [20:47:19.967] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 10/10) 
## INFO  [20:47:19.976] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 1/10) 
## INFO  [20:47:19.994] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 8/10) 
## INFO  [20:47:20.002] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 10/10) 
## INFO  [20:47:20.019] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 4/10) 
## INFO  [20:47:20.027] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 9/10) 
## INFO  [20:47:20.103] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 8/10) 
## INFO  [20:47:20.113] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 3/10) 
## INFO  [20:47:20.189] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 1/10) 
## INFO  [20:47:20.379] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 4/10) 
## INFO  [20:47:20.397] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 6/10) 
## INFO  [20:47:20.423] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 7/10) 
## INFO  [20:47:20.440] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 5/10) 
## INFO  [20:47:20.448] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 10/10) 
## INFO  [20:47:20.456] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 6/10) 
## INFO  [20:47:20.473] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 3/10) 
## INFO  [20:47:20.703] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 3/10) 
## INFO  [20:47:20.714] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 6/10) 
## INFO  [20:47:20.731] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 1/1) 
## INFO  [20:47:20.738] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 7/10) 
## INFO  [20:47:20.748] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 9/10) 
## INFO  [20:47:20.794] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 5/10) 
## INFO  [20:47:20.989] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 1/1) 
## INFO  [20:47:21.006] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 2/10) 
## INFO  [20:47:21.024] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 4/10) 
## INFO  [20:47:21.225] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 1/10) 
## INFO  [20:47:21.234] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 9/10) 
## INFO  [20:47:22.618] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 1/10) 
## INFO  [20:47:22.695] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 9/10) 
## INFO  [20:47:22.704] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 1/10) 
## INFO  [20:47:24.109] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 4/10) 
## INFO  [20:47:24.117] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 2/10) 
## INFO  [20:47:25.675] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 8/10) 
## INFO  [20:47:25.726] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 3/10) 
## INFO  [20:47:27.115] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 1/1) 
## INFO  [20:47:28.155] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 5/10) 
## INFO  [20:47:28.165] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 3/10) 
## INFO  [20:47:28.186] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 6/10) 
## INFO  [20:47:28.233] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 10/10) 
## INFO  [20:47:28.458] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 7/10) 
## INFO  [20:47:29.832] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 6/10) 
## INFO  [20:47:29.841] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 5/10) 
## INFO  [20:47:29.859] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 3/10) 
## INFO  [20:47:29.878] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 2/10) 
## INFO  [20:47:29.898] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 7/10) 
## INFO  [20:47:29.950] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 10/10) 
## INFO  [20:47:31.332] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 9/10) 
## INFO  [20:47:31.342] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 8/10) 
## INFO  [20:47:31.360] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 10/10) 
## INFO  [20:47:31.439] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 2/10) 
## INFO  [20:47:31.513] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 4/10) 
## INFO  [20:47:32.917] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 7/10) 
## INFO  [20:47:32.994] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 8/10) 
## INFO  [20:47:33.003] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 3/10) 
## INFO  [20:47:33.194] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 1/10) 
## INFO  [20:47:33.212] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 2/10) 
## INFO  [20:47:33.221] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 10/10) 
## INFO  [20:47:33.495] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 8/10) 
## INFO  [20:47:33.512] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 9/10) 
## INFO  [20:47:33.704] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 4/10) 
## INFO  [20:47:33.753] [mlr3] Applying learner 'classif.ranger' on task 'spam' (iter 6/10) 
## INFO  [20:47:35.136] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 10/10) 
## INFO  [20:47:35.147] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 6/10) 
## INFO  [20:47:35.332] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 5/10) 
## INFO  [20:47:35.380] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 7/10) 
## INFO  [20:47:35.581] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 1/1) 
## INFO  [20:47:35.643] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 2/10) 
## INFO  [20:47:35.653] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 1/1) 
## INFO  [20:47:35.826] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 7/10) 
## INFO  [20:47:35.835] [mlr3] Applying learner 'classif.ranger' on task 'sonar' (iter 5/10) 
## INFO  [20:47:35.910] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 1/1) 
## INFO  [20:47:35.951] [mlr3] Applying learner 'classif.rpart' on task 'german_credit' (iter 9/10) 
## INFO  [20:47:35.969] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 5/10) 
## INFO  [20:47:35.980] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 1/1) 
## INFO  [20:47:35.997] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 4/10) 
## INFO  [20:47:36.257] [mlr3] Applying learner 'classif.featureless' on task 'sonar' (iter 1/1) 
## INFO  [20:47:36.264] [mlr3] Applying learner 'classif.featureless' on task 'spam' (iter 2/10) 
## INFO  [20:47:36.274] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 1/10) 
## INFO  [20:47:36.322] [mlr3] Applying learner 'classif.rpart' on task 'spam' (iter 2/10) 
## INFO  [20:47:36.366] [mlr3] Applying learner 'classif.featureless' on task 'german_credit' (iter 7/10) 
## INFO  [20:47:36.375] [mlr3] Applying learner 'classif.rpart' on task 'sonar' (iter 9/10) 
## INFO  [20:47:36.414] [mlr3] Finished benchmark

查看模型的表現(xiàn)，使用多種度量指標(biāo)：

measures <- msrs(c("classif.acc", "classif.mcc"))

tab <- bmr$aggregate(measures)
print(tab)
##     nr      resample_result       task_id          learner_id resampling_id
##  1:  1 <ResampleResult[22]>          spam      classif.ranger       holdout
##  2:  2 <ResampleResult[22]>          spam      classif.ranger            cv
##  3:  3 <ResampleResult[22]>          spam       classif.rpart       holdout
##  4:  4 <ResampleResult[22]>          spam       classif.rpart            cv
##  5:  5 <ResampleResult[22]>          spam classif.featureless       holdout
##  6:  6 <ResampleResult[22]>          spam classif.featureless            cv
##  7:  7 <ResampleResult[22]> german_credit      classif.ranger       holdout
##  8:  8 <ResampleResult[22]> german_credit      classif.ranger            cv
##  9:  9 <ResampleResult[22]> german_credit       classif.rpart       holdout
## 10: 10 <ResampleResult[22]> german_credit       classif.rpart            cv
## 11: 11 <ResampleResult[22]> german_credit classif.featureless       holdout
## 12: 12 <ResampleResult[22]> german_credit classif.featureless            cv
## 13: 13 <ResampleResult[22]>         sonar      classif.ranger       holdout
## 14: 14 <ResampleResult[22]>         sonar      classif.ranger            cv
## 15: 15 <ResampleResult[22]>         sonar       classif.rpart       holdout
## 16: 16 <ResampleResult[22]>         sonar       classif.rpart            cv
## 17: 17 <ResampleResult[22]>         sonar classif.featureless       holdout
## 18: 18 <ResampleResult[22]>         sonar classif.featureless            cv
##     iters classif.acc classif.mcc
##  1:     1   0.9445893   0.8835453
##  2:    10   0.9495723   0.8943582
##  3:     1   0.8917862   0.7725102
##  4:    10   0.8934967   0.7765629
##  5:     1   0.6069100   0.0000000
##  6:    10   0.6059511   0.0000000
##  7:     1   0.7567568   0.4358851
##  8:    10   0.7670000   0.3927548
##  9:     1   0.6996997   0.2847394
## 10:    10   0.7290000   0.2984376
## 11:     1   0.6516517   0.0000000
## 12:    10   0.7000000   0.0000000
## 13:     1   0.7971014   0.6247458
## 14:    10   0.8221429   0.6390361
## 15:     1   0.6956522   0.3981439
## 16:    10   0.6545238   0.3098052
## 17:     1   0.4782609   0.0000000
## 18:    10   0.5340476   0.0000000

可視化結(jié)果

library(ggplot2)
autoplot(bmr) + theme_bw() + 
  theme(axis.text.x = element_text(angle = 45,hjust = 1))

上面的圖給出了多個(gè)模型在不同數(shù)據(jù)集中的平均表現(xiàn)邢滑，我們也可以查看多個(gè)模型在某一個(gè)特定數(shù)據(jù)集中的表現(xiàn)：

bmr_german <- bmr$clone(deep = T)$filter(task_ids = "german_credit",resampling_ids = "holdout")
autoplot(bmr_german, type = "roc")

當(dāng)然也可以只提取其中一個(gè)結(jié)果：

tab <- bmr$aggregate(measures)
rr <- tab[task_id == "german_credit" & learner_id ==  "classif.ranger"]$resample_result[[1]]
print(rr)
## <ResampleResult> of 1 iterations
## * Task: german_credit
## * Learner: classif.ranger
## * Warnings: 0 in 0 iterations
## * Errors: 0 in 0 iterations

查看一個(gè)結(jié)果的表現(xiàn)：

rr$aggregate(msr("classif.auc"))
## classif.auc 
##   0.8085969

合并多個(gè)BenchmarkResult腐螟，比如在2臺(tái)電腦上做了2個(gè)不同的benchmarks，可以直接合并成一個(gè)更大的對象：

task <- tsk("iris")
resampling <- rsmp("holdout")$instantiate(task)

rr1 <- resample(task, lrn("classif.rpart"), resampling)
## INFO  [20:47:40.585] [mlr3] Applying learner 'classif.rpart' on task 'iris' (iter 1/1)
rr2 <- resample(task, lrn("classif.featureless"), resampling)
## INFO  [20:47:40.606] [mlr3] Applying learner 'classif.featureless' on task 'iris' (iter 1/1)

# 通過以下代碼合并結(jié)果
bmr1 <- as_benchmark_result(rr1)
bmr2 <- as_benchmark_result(rr2)

bmr1$combine(bmr2)

bmr1
## <BenchmarkResult> of 2 rows with 2 resampling runs
##  nr task_id          learner_id resampling_id iters warnings errors
##   1    iris       classif.rpart       holdout     1        0      0
##   2    iris classif.featureless       holdout     1        0      0

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者

人面猴
序言：七十年代末困后，一起剝皮案震驚了整個(gè)濱河市乐纸，隨后出現(xiàn)的幾起案子，更是在濱河造成了極大的恐慌摇予，老刑警劉巖汽绢，帶你破解...
沈念sama閱讀 206,013評論 6贊 481
死咒
序言：濱河連續(xù)發(fā)生了三起死亡事件，死亡現(xiàn)場離奇詭異侧戴，居然都是意外死亡宁昭，警方通過查閱死者的電腦和手機(jī)跌宛，發(fā)現(xiàn)死者居然都...
沈念sama閱讀 88,205評論 2贊 382
救了他兩次的神仙讓他今天三更去死
文/潘曉璐我一進(jìn)店門，熙熙樓的掌柜王于貴愁眉苦臉地迎上來积仗，“玉大人疆拘，你說我怎么就攤上這事〖挪埽” “怎么了哎迄？”我有些...
開封第一講書人閱讀 152,370評論 0贊 342
道士緝兇錄：失蹤的賣姜人
文/不壞的土叔我叫張陵，是天一觀的道長稀颁。經(jīng)常有香客問我芬失，道長，這世上最難降的妖魔是什么匾灶？我笑而不...
開封第一講書人閱讀 55,168評論 1贊 278
?港島之戀（遺憾婚禮）
正文為了忘掉前任棱烂，我火速辦了婚禮，結(jié)果婚禮上阶女，老公的妹妹穿的比我還像新娘颊糜。我一直安慰自己，他們只是感情好秃踩，可當(dāng)我...
茶點(diǎn)故事閱讀 64,153評論 5贊 371
惡毒庶女頂嫁案：這布局不是一般人想出來的
文/花漫我一把揭開白布衬鱼。她就那樣靜靜地躺著，像睡著了一般憔杨。火紅的嫁衣襯著肌膚如雪鸟赫。梳的紋絲不亂的頭發(fā)上，一...
開封第一講書人閱讀 48,954評論 1贊 283
城市分裂傳說
那天消别，我揣著相機(jī)與錄音抛蚤，去河邊找鬼。笑死寻狂，一個(gè)胖子當(dāng)著我的面吹牛岁经，可吹牛的內(nèi)容都是我干的。我是一名探鬼主播蛇券，決...
沈念sama閱讀 38,271評論 3贊 399
雙鴛鴦連環(huán)套：你想象不到人心有多黑
文/蒼蘭香墨我猛地睜開眼缀壤，長吁一口氣：“原來是場噩夢啊……” “哼！你這毒婦竟也來了纠亚？” 一聲冷哼從身側(cè)響起塘慕，我...
開封第一講書人閱讀 36,916評論 0贊 259
萬榮殺人案實(shí)錄
序言：老撾萬榮一對情侶失蹤，失蹤者是張志新（化名）和其女友劉穎菜枷，沒想到半個(gè)月后苍糠，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體，經(jīng)...
沈念sama閱讀 43,382評論 1贊 300
?護(hù)林員之死
正文獨(dú)居荒郊野嶺守林人離奇死亡啤誊，尸身上長有42處帶血的膿包…… 初始之章·張勛以下內(nèi)容為張勛視角年9月15日...
茶點(diǎn)故事閱讀 35,877評論 2贊 323
?白月光啟示錄
正文我和宋清朗相戀三年岳瞭，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了拥娄。大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
茶點(diǎn)故事閱讀 37,989評論 1贊 333
活死人
序言：一個(gè)原本活蹦亂跳的男人離奇死亡瞳筏，死狀恐怖稚瘾，靈堂內(nèi)的尸體忽然破棺而出，到底是詐尸還是另有隱情姚炕，我是刑警寧澤摊欠，帶...
沈念sama閱讀 33,624評論 4贊 322
?日本核電站爆炸內(nèi)幕
正文年R本政府宣布，位于F島的核電站柱宦，受9級特大地震影響些椒，放射性物質(zhì)發(fā)生泄漏。R本人自食惡果不足惜掸刊，卻給世界環(huán)境...
茶點(diǎn)故事閱讀 39,209評論 3贊 307
男人毒藥：我在死后第九天來索命
文/蒙蒙一免糕、第九天我趴在偏房一處隱蔽的房頂上張望。院中可真熱鬧忧侧，春花似錦石窑、人聲如沸。這莊子的主人今日做“春日...
開封第一講書人閱讀 30,199評論 0贊 19
一樁弒父案松逊，背后竟有這般陰謀
文/蒼蘭香墨我抬頭看了看天上的太陽。三九已至肯夏，卻和暖如春经宏，著一層夾襖步出監(jiān)牢的瞬間，已是汗流浹背驯击。一陣腳步聲響...
開封第一講書人閱讀 31,418評論 1贊 260
情欲美人皮
我被黑心中介騙來泰國打工烛恤，沒想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留，地道東北人余耽。一個(gè)月前我還...
沈念sama閱讀 45,401評論 2贊 352
代替公主和親
正文我出身青樓，卻偏偏與公主長得像苹熏，于是被迫代替她去往敵國和親碟贾。傳聞我的和親對象是個(gè)殘疾皇子，可洞房花燭夜當(dāng)晚...
茶點(diǎn)故事閱讀 42,700評論 2贊 345

R機(jī)器學(xué)習(xí)mlr3：模型評價(jià)和比較

二分類變量和ROC曲線

重抽樣

benchmark

推薦閱讀更多精彩內(nèi)容