Feature Selection
Ensemble Learning
- bagging method and boosting method
Bagging
- sampling with replacement
- decrease variance by introducing randomness into your model framework
- random forest = bagging + decision tree
Random Forest
- description of random forest
there are n samples with m features in the training data- take n observations with replacement each time 行上有放回的抽樣
- in these n observations, take k features (k < m) to calculate a best decision tree 列上抽樣不放回
- repeat the below steps for several times, combine all the decision tree to a random forest
- features:
- decrease variance by introducing randomness into the model framework
- the destributions of each n observations are the same as the original training data
- we don't need to do pruning for each "weak" decision tree
- less overfitting
- parallel implementation
- Feature Importance value in Random Forest
(advance topic: out of bag evaluation)
how to define performance? let the column of the feature be a list of random numbers and calculate the output by the model, get the loss- not negative nor positive, just show how much a special feature influences the model
Support Vector Machine
-
SVM: maximize the minimum margin
image.png
image.png
if not linear, add noise factors into the equation to map it to a higher dimension space, applying kernel functionimage.png
Why Feature Selection?
- reduce overfitting
- better understanding your model
- improve model stability (i.e. improve generalization)
取決于你想要做什么渊额，如果是做一個調(diào)查，想研究每一個feature的貢獻(xiàn)，則需要刪除一些data以減少相關(guān)性太大的features對模型的影響；如果想要做prediction，則只關(guān)心結(jié)果是否準(zhǔn)確痊硕，不太需要刪除features。模型穩(wěn)定性差：某一個feature變化一點點而導(dǎo)致系數(shù)變化特別大，說明模型不穩(wěn)定variance特別大盟萨，原因可能是model特別復(fù)雜或者相關(guān)性features太多。解決辦法最直觀的：regularization
Pearson Correlation
to measrue linear dependency between features
-
means covariance and
means standard deviation
- covariance:
where
Regularization Models
L1 tends to provide sparse solution
L2 tends to spread out more equality
image.png
