【前言】Drop Path是NAS中常用到的一種正則化方法痘番，由于網(wǎng)絡(luò)訓練的過程中常常是動態(tài)的醉冤，Drop Path就成了一個不錯的正則化工具蓖宦，在FractalNet叁执、NASNet等都有廣泛使用茄厘。
Dropout
Dropout是最早的用于解決過擬合的方法，是所有drop類方法的大前輩谈宛。Dropout在12年被Hinton提出次哈，并且在ImageNet Classification with Deep Convolutional Neural Network工作AlexNet中使用到了Dropout。
原理 ：在前向傳播的時候吆录，讓某個神經(jīng)元激活以概率1-keep_prob（0<p<1）停止工作窑滞。
功能 ： 這樣可以讓模型泛化能力更強，因為其不會過于以來某些局部的節(jié)點恢筝。訓練階段以keep_prob的概率保留哀卫，以1-keep_prob的概率關(guān)閉；測試階段所有的神經(jīng)元都不關(guān)閉撬槽，但是對訓練階段應(yīng)用了dropout的神經(jīng)元此改，輸出值需要乘以keep_prob。
具體是這樣的：
假設(shè)一個神經(jīng)元的輸出激活值為
a侄柔，在不使用dropout的情況下共啃，其輸出期望值為a，如果使用了dropout暂题，神經(jīng)元就可能有保留和關(guān)閉兩種狀態(tài)移剪，把它看作一個離散型隨機變量，它就符合概率論中的0-1分布薪者，其輸出激活值的期望變?yōu)?p*a+(1-p)*0=pa纵苛，此時若要保持期望和不使用dropout時一致，就要除以p。 作者：種子_fe 鏈接：https://www.imooc.com/article/30129
實現(xiàn) ： pytorch中的實現(xiàn)如下攻人。
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="python" cid="n13" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 1.125rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(255, 255, 255); position: relative !important; color: rgb(122, 122, 122); padding: 0.5rem 1.125em; margin-bottom: 0.88em; border: 1px solid rgb(122, 122, 122); line-height: 1.5rem; width: inherit; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">class _DropoutNd(Module):
constants = ['p', 'inplace']
p: float
inplace: bool
def init(self, p: float = 0.5, inplace: bool = False) -> None:
super(_DropoutNd, self).init()
if p < 0 or p > 1:
raise ValueError("dropout probability has to be between 0 and 1, "
"but got {}".format(p))
self.p = p
self.inplace = inplace
def extra_repr(self) -> str:
return 'p={}, inplace={}'.format(self.p, self.inplace)
class Dropout(_DropoutNd):
def forward(self, input: Tensor) -> Tensor:
return F.dropout(input, self.p, self.training, self.inplace)</pre>
funtional.py中的dropout實現(xiàn)：
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="python" cid="n15" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 1.125rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(255, 255, 255); position: relative !important; color: rgb(122, 122, 122); padding: 0.5rem 1.125em; margin-bottom: 0.88em; border: 1px solid rgb(122, 122, 122); line-height: 1.5rem; width: inherit; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">def dropout(input: Tensor, p: float = 0.5, training: bool = True, inplace: bool = False) -> Tensor:
r"""
During training, randomly zeroes some of the elements of the input
tensor with probability :attr:p using samples from a Bernoulli
distribution.
See :class:~torch.nn.Dropout for details.
Args:
p: probability of an element to be zeroed. Default: 0.5
training: apply dropout if is True. Default: True
inplace: If set to True, will do this operation in-place. Default: False
"""
if has_torch_function_unary(input):
return handle_torch_function(dropout, (input,), input, p=p, training=training, inplace=inplace)
if p < 0.0 or p > 1.0:
raise ValueError("dropout probability has to be between 0 and 1, " "but got {}".format(p))
return VF.dropout(input, p, training) if inplace else _VF.dropout(input, p, training)</pre>
最終在Dropout.cpp中找到具體實現(xiàn)：
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="c++" cid="n17" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 1.125rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(255, 255, 255); position: relative !important; color: rgb(122, 122, 122); padding: 0.5rem 1.125em; margin-bottom: 0.88em; border: 1px solid rgb(122, 122, 122); line-height: 1.5rem; width: inherit; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">template<bool feature_dropout, bool alpha_dropout, bool inplace, typename T>
Ctype<inplace> _dropout_impl(T& input, double p, bool train) {
TORCH_CHECK(p >= 0 && p <= 1, "dropout probability has to be between 0 and 1, but got ", p);
if (p == 0 || !train || input.numel() == 0) {
return input;
}
if (p == 1) {
return multiply<inplace>(input, at::zeros({}, input.options()));
}
at::Tensor b; // used for alpha_dropout only
auto noise = feature_dropout ? make_feature_noise(input) : at::empty_like(input, LEGACY_CONTIGUOUS_MEMORY_FORMAT);
noise.bernoulli_(1 - p);
if (alpha_dropout) {
constexpr double alpha = 1.7580993408473766;
double a = 1. / std::sqrt((alpha * alpha * p + 1) * (1 - p));
b = noise.add(-1).mul_(alpha * a).add_(alpha * a * p);
noise.mul_(a);
} else {
noise.div_(1 - p);
}
if (!alpha_dropout) {
return multiply<inplace>(input, noise);
} else {
return multiply<inplace>(input, noise).add_(b);
}
}</pre>
流程：
判斷p的范圍 以及訓練狀態(tài)
使用1-p的概率得到伯努利分布（0-1分布）
(input / 1-p) * 伯努利分布
Drop Path
原理 ：字如其名幔虏，Drop Path就是隨機將深度學習網(wǎng)絡(luò)中的多分支結(jié)構(gòu)隨機刪除。
功能 ：一般可以作為正則化手段加入網(wǎng)絡(luò)贝椿，但是會增加網(wǎng)絡(luò)訓練的難度想括。尤其是在NAS問題中，如果設(shè)置的drop prob過高烙博，模型甚至有可能不收斂瑟蜈。
實現(xiàn) ：
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="python" cid="n31" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 1.125rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(255, 255, 255); position: relative !important; color: rgb(122, 122, 122); padding: 0.5rem 1.125em; margin-bottom: 0.88em; border: 1px solid rgb(122, 122, 122); line-height: 1.5rem; width: inherit; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">def drop_path(x, drop_prob: float = 0., training: bool = False):
if drop_prob == 0. or not training:
return x
keep_prob = 1 - drop_prob
shape = (x.shape[0],) + (1,) * (x.ndim - 1) # work with diff dim tensors, not just 2D ConvNets
random_tensor = keep_prob + torch.rand(shape, dtype=x.dtype, device=x.device)
random_tensor.floor_() # binarize
output = x.div(keep_prob) * random_tensor
return output
class DropPath(nn.Module):
"""Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
"""
def init(self, drop_prob=None):
super(DropPath, self).init()
self.drop_prob = drop_prob
def forward(self, x):
return drop_path(x, self.drop_prob, self.training)</pre>
有了Dropout的理論鋪墊，這里的實現(xiàn)就比較明了了渣窜，具體使用的時候一般是這樣的：
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="python" cid="n33" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 1.125rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(255, 255, 255); position: relative !important; color: rgb(122, 122, 122); padding: 0.5rem 1.125em; margin-bottom: 0.88em; border: 1px solid rgb(122, 122, 122); line-height: 1.5rem; width: inherit; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">x = x + self.drop_path(self.conv(x))</pre>
Drop Path不能直接這樣使用：
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="python" cid="n35" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 1.125rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(255, 255, 255); position: relative !important; color: rgb(122, 122, 122); padding: 0.5rem 1.125em; margin-bottom: 0.88em; border: 1px solid rgb(122, 122, 122); line-height: 1.5rem; width: inherit; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">x = self.drop_path(x)</pre>
Reference
https://www.cnblogs.com/dan-baishucaizi/p/14703263.html
