Logistic Regression
在線性回歸中菇存，我們尋找的連續(xù)型隨機(jī)變量
和
的函數(shù)關(guān)系式為：
夸研，其中
為待估參數(shù)（包含截距項(xiàng)
，即%5ET)
依鸥，
）亥至，
為隨機(jī)誤差。那么贱迟，如果是離散型隨機(jī)變量姐扮，例如服從Bernoulli分布、多項(xiàng)分布等衣吠，又應(yīng)該怎么樣來(lái)描述和的關(guān)系呢茶敏？
下面我們只討論一般的Logistic Regression，即)
缚俏。直觀的想法是假設(shè)下面關(guān)系成立：
其中
惊搏。然而，這樣就會(huì)出現(xiàn)一個(gè)問(wèn)題忧换，
與
應(yīng)該是在![[0,1]](https://math.jianshu.com/math?formula=%5B0%2C1%5D)
范圍內(nèi)恬惯。因此，我們可以考慮做以下修正包雀，將其映射到范圍內(nèi)：
即先取指數(shù)宿崭，映射到非負(fù)區(qū)間，再做歸一化才写。對(duì)上式再做簡(jiǎn)單化簡(jiǎn)葡兑，就可以得到Logistics Regression的常見(jiàn)形式：
因此，上述變換等價(jià)于對(duì)
做了sigmoid變換赞草，對(duì)應(yīng)的sigmoid函數(shù)為%3D%5Cdfrac%7B1%7D%7B1%2Be%5E%7B-x%7D%7D)
讹堤；在多分類的情形下，即為softmax變換厨疙；在其他情形下洲守，還有其他對(duì)應(yīng)的函數(shù)，感興趣的讀者可以參考以下Generalized Model的Mean Function沾凄。
注1：這里我們只是給出了一個(gè)容易理解的方式梗醇，為什么這個(gè)映射函數(shù)恰好是)
而不是其他函數(shù)？其實(shí)是可以從凸優(yōu)化問(wèn)題中利用KKT條件求解出的撒蟀，詳見(jiàn)論文The equivalence of logistic regression and maximum entropy models叙谨。
的極大似然估計(jì)
在給定樣本%2Ci%3D1%2C2%2C%5Ccdots%2Cn%5C%7D)
的情況下，首先我們需要寫出似然函數(shù)保屯。由于手负，因此的分布函數(shù)為%3Dp%5Ey(1-p)%5E%7B1-y%7D)
涤垫。似然函數(shù)為
%26%3D%5Cprod%5Climits_%7Bi%3D1%7D%5Enf_Y(Y_i%3B%5Cbeta)%5C%5C%0A%26%3D%5Cprod%5Climits_%7Bi%3D1%7D%5Enp_i%5E%7BY_i%7D(1-p_i)%5E%7B1-Y_i%7D%5C%5C%0A%26%3Dp_i%5E%7B%5Csum%5Climits_%7Bi%3D1%7D%5EnY_i%7D%20(1-p_i)%5E%7B%5Csum%5Climits_%7Bi%3D1%7D%5En(1-Y_i)%7D%0A%5Cend%7Balign%7D%5C%5C)
取對(duì)數(shù)之后，得到
%26%3D%5Clog%20L(%5Cbeta%3BX_i%2CY_i%2Ci%3D1%2C2%2C%5Ccdots%2Cn)%5C%5C%26%3D%20%5Csum%5Climits_%7Bi%3D1%7D%5EnY_i%5Clog%20p_i%20%2B%20%20%5Csum%5Climits_%7Bi%3D1%7D%5En(1-Y_i)%5Clog(1-%20p_i)%5Cend%7Balign%7D%5C%5C)
注意到我們?cè)谇耙还?jié)已經(jīng)假定了Logistic模型竟终，即)
蝠猬，為了保持形式的簡(jiǎn)潔，在上式中仍用
代替统捶。注意到
%5E2%7D%5Ccdot%20e%5E%7B-%5Cbeta%5ETX%7D%5Ccdot%20(-X)%5C%5C%0A%26%3D%5Cdfrac%7BXe%5E%7B-%5Cbeta%5ETX%7D%7D%7B(1%2Be%5E%7B-%5Cbeta%5ETX%7D)%5E2%7D%5C%5C%0A%26%3DXp(1-p)%0A%5Cend%7Balign%7D%5C%5C)
對(duì)log似然函數(shù)求偏導(dǎo)榆芦，并令其為0：![\begin{align}\dfrac{\partial l}{\partial \beta}&=\sum\limits_{i=1}^nY_i \dfrac{1}{p_i}\cdot\dfrac{\partial p_i}{\partial \beta}-\sum\limits_{i=1}^n(1-Y_i) \dfrac{1}{1-p_i}\cdot\dfrac{\partial p_i}{\partial \beta}\\&= \sum\limits_{i=1}^n \dfrac{X_iY_ie^{-\beta^TX_i}}{1+e^{-\beta^TX_i}}-\sum\limits_{i=1}^n \dfrac{X_i(1-Y_i)}{1+e^{-\beta^TX_i}}\\&=\sum\limits_{i=1}^n \dfrac{X_i}{1+e^{-\beta^TX_i}}[-1+(1+e^{-\beta^TX_i})Y_i]\\&=\sum\limits_{\{i|Y_i=1\}} \dfrac{X_i}{1+e^{-\beta^TX_i}}e^{-\beta^TX_i}-\sum\limits_{\{i|Y_i=0\}} \dfrac{X_i}{1+e^{-\beta^TX_i}}\\&=\sum\limits_{\{i|Y_i=1\}} X_i-\sum\limits_{i=1}^n \dfrac{X_i}{1+e^{-\beta^TX_i}}\\&=0\end{align}\\](https://math.jianshu.com/math?formula=%5Cbegin%7Balign%7D%0A%5Cdfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Cbeta%7D%26%3D%5Csum%5Climits_%7Bi%3D1%7D%5EnY_i%20%5Cdfrac%7B1%7D%7Bp_i%7D%5Ccdot%5Cdfrac%7B%5Cpartial%20p_i%7D%7B%5Cpartial%20%5Cbeta%7D-%5Csum%5Climits_%7Bi%3D1%7D%5En(1-Y_i)%20%5Cdfrac%7B1%7D%7B1-p_i%7D%5Ccdot%5Cdfrac%7B%5Cpartial%20p_i%7D%7B%5Cpartial%20%5Cbeta%7D%5C%5C%0A%26%3D%20%5Csum%5Climits_%7Bi%3D1%7D%5En%20%5Cdfrac%7BX_iY_ie%5E%7B-%5Cbeta%5ETX_i%7D%7D%7B1%2Be%5E%7B-%5Cbeta%5ETX_i%7D%7D-%5Csum%5Climits_%7Bi%3D1%7D%5En%20%5Cdfrac%7BX_i(1-Y_i)%7D%7B1%2Be%5E%7B-%5Cbeta%5ETX_i%7D%7D%5C%5C%0A%26%3D%5Csum%5Climits_%7Bi%3D1%7D%5En%20%5Cdfrac%7BX_i%7D%7B1%2Be%5E%7B-%5Cbeta%5ETX_i%7D%7D%5B-1%2B(1%2Be%5E%7B-%5Cbeta%5ETX_i%7D)Y_i%5D%5C%5C%0A%26%3D%5Csum%5Climits_%7B%5C%7Bi%7CY_i%3D1%5C%7D%7D%20%5Cdfrac%7BX_i%7D%7B1%2Be%5E%7B-%5Cbeta%5ETX_i%7D%7De%5E%7B-%5Cbeta%5ETX_i%7D-%5Csum%5Climits_%7B%5C%7Bi%7CY_i%3D0%5C%7D%7D%20%5Cdfrac%7BX_i%7D%7B1%2Be%5E%7B-%5Cbeta%5ETX_i%7D%7D%5C%5C%0A%26%3D%5Csum%5Climits_%7B%5C%7Bi%7CY_i%3D1%5C%7D%7D%20X_i-%5Csum%5Climits_%7Bi%3D1%7D%5En%20%5Cdfrac%7BX_i%7D%7B1%2Be%5E%7B-%5Cbeta%5ETX_i%7D%7D%5C%5C%0A%26%3D0%0A%5Cend%7Balign%7D%5C%5C)
上述方程沒(méi)有顯式解，一般只能用牛頓迭代法求最大似然估計(jì)的數(shù)值解瘾境。
Logistic Loss
我們重新化簡(jiǎn)一下對(duì)數(shù)似然函數(shù)歧杏，
![\begin{align}
l(\beta;X_i,Y_i,i=1,2,\cdots,n)&= \sum\limits_{i=1}^nY_i\log p_i + \sum\limits_{i=1}^n(1-Y_i)\log(1- p_i)\\&= -\sum\limits_{i=1}^nY_i\log(1+e^{-\beta^TX_i})+ \sum\limits_{i=1}^n(1-Y_i)[-\beta^TX_i-\log(1+e^{-\beta^TX_i})]\\&=\sum\limits_{i=1}^n[(Y_i-1)\beta^TX_i-\log(1+e^{-\beta^TX_i})]\\&=\sum\limits_{i=1}^n\left[(Y_i-1)\beta^TX_i+\log\left(\frac{e^{\beta^TX_i}}{1+e^{\beta^TX_i}}\right)\right]\\&=\sum\limits_{i=1}^n\left[Y_i\beta^TX_i-\log\left(1+e^{\beta^TX_i}\right)\right]\end{align}\\](https://math.jianshu.com/math?formula=%5Cbegin%7Balign%7D%0Al(%5Cbeta%3BX_i%2CY_i%2Ci%3D1%2C2%2C%5Ccdots%2Cn)%26%3D%20%5Csum%5Climits_%7Bi%3D1%7D%5EnY_i%5Clog%20p_i%20%2B%20%20%5Csum%5Climits_%7Bi%3D1%7D%5En(1-Y_i)%5Clog(1-%20p_i)%5C%5C%26%3D%20-%5Csum%5Climits_%7Bi%3D1%7D%5EnY_i%5Clog(1%2Be%5E%7B-%5Cbeta%5ETX_i%7D)%2B%20%20%5Csum%5Climits_%7Bi%3D1%7D%5En(1-Y_i)%5B-%5Cbeta%5ETX_i-%5Clog(1%2Be%5E%7B-%5Cbeta%5ETX_i%7D)%5D%5C%5C%26%3D%5Csum%5Climits_%7Bi%3D1%7D%5En%5B(Y_i-1)%5Cbeta%5ETX_i-%5Clog(1%2Be%5E%7B-%5Cbeta%5ETX_i%7D)%5D%5C%5C%26%3D%5Csum%5Climits_%7Bi%3D1%7D%5En%5Cleft%5B(Y_i-1)%5Cbeta%5ETX_i%2B%5Clog%5Cleft(%5Cfrac%7Be%5E%7B%5Cbeta%5ETX_i%7D%7D%7B1%2Be%5E%7B%5Cbeta%5ETX_i%7D%7D%5Cright)%5Cright%5D%5C%5C%26%3D%5Csum%5Climits_%7Bi%3D1%7D%5En%5Cleft%5BY_i%5Cbeta%5ETX_i-%5Clog%5Cleft(1%2Be%5E%7B%5Cbeta%5ETX_i%7D%5Cright)%5Cright%5D%5Cend%7Balign%7D%5C%5C)
因此，我們有
![\begin{align}\max\limits_{\beta}l(\beta;X_i,Y_i,i=1,2,\cdots,n)&=\max\limits_{\beta}\sum\limits_{i=1}^n\left[Y_i\beta^TX_i-\log\left(1+e^{\beta^TX_i}\right)\right]\\&=\min\limits_{\beta}\sum\limits_{i=1}^n\left[-Y_i\beta^TX_i+\log\left(1+e^{\beta^TX_i}\right)\right]\\LogisticLoss(X_i,Y_i;\beta)&\triangleq -Y_i\beta^TX_i+\log\left(1+e^{\beta^TX_i}\right)\end{align}\\](https://math.jianshu.com/math?formula=%5Cbegin%7Balign%7D%0A%5Cmax%5Climits_%7B%5Cbeta%7Dl(%5Cbeta%3BX_i%2CY_i%2Ci%3D1%2C2%2C%5Ccdots%2Cn)%26%3D%0A%5Cmax%5Climits_%7B%5Cbeta%7D%5Csum%5Climits_%7Bi%3D1%7D%5En%5Cleft%5BY_i%5Cbeta%5ETX_i-%5Clog%5Cleft(1%2Be%5E%7B%5Cbeta%5ETX_i%7D%5Cright)%5Cright%5D%5C%5C%0A%26%3D%5Cmin%5Climits_%7B%5Cbeta%7D%5Csum%5Climits_%7Bi%3D1%7D%5En%5Cleft%5B-Y_i%5Cbeta%5ETX_i%2B%5Clog%5Cleft(1%2Be%5E%7B%5Cbeta%5ETX_i%7D%5Cright)%5Cright%5D%5C%5C%0ALogisticLoss(X_i%2CY_i%3B%5Cbeta)%26%5Ctriangleq%20-Y_i%5Cbeta%5ETX_i%2B%5Clog%5Cleft(1%2Be%5E%7B%5Cbeta%5ETX_i%7D%5Cright)%0A%5Cend%7Balign%7D%5C%5C)
注2：在實(shí)際計(jì)算中迷守，通常還要除以樣本數(shù)
犬绒，控制梯度大小，因?yàn)橛?jì)算
的時(shí)候是根據(jù)gradient-based算法兑凿。
注3：我們討論的是
的情況凯力，在
時(shí)，Logistic Loss有不同的形式（詳見(jiàn)Which loss function is correct for logistic regression?）礼华，
%3D%5Clog(1%2Be%5E%7B-Y_i%5Cbeta%5ETX_i%7D)%5C%5C)
而后者的標(biāo)簽與Adaboost推導(dǎo)時(shí)默認(rèn)的標(biāo)簽相同咐鹤，Logistic Loss與Adaboost的Exponential Loss也有一定相似性，在底數(shù)為
的情況下圣絮，%5Cleq%20x%2C%5C%20x%5Cgeq%20-1)
祈惶，Logistic Loss的圖像在Exponential Loss下方，甚至還可以將Adaboost修改為優(yōu)化Logistic Loss（詳見(jiàn)Logistic Regression）扮匠。
A plot of the exponential loss, the logistic loss (using both logarithm base 2 and logarithm base?e), and classification loss.
最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
