curl的奇特問題
最近在使用curl的時候發(fā)現(xiàn)一個奇怪的問題。使用curl訪問sftp服務器時總是報錯心例。我執(zhí)行的命令如下:
curl -u sftptest:passwd sftp://127.0.0.1:22/home/sftptest/
然后返回了一大篇html, 主要的信息如下:
...
<title>ERROR: The requested URL could not be retrieved</title>
...
<p><b>Unsupported Request Method and Protocol</b></p>
首先需要確認的是sftp是否在curl中開啟了。執(zhí)行curl -V發(fā)現(xiàn)確實支持curl. 接下來打開verbose模式弦追，發(fā)現(xiàn)了一些有趣的信息:
* Trying 172.17.10.80...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to (nil) (172.17.18.84) port 8080 (#0)
* Server auth using Basic with user 'sftptest'
> GET sftp://sftptest:password@127.0.0.1/home/sftptest HTTP/1.1
> Host: 127.0.0.1:22
> Authorization: Basic c2Z0cHRlc3Q6MXEydzNlNHI=
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.52.1 OpenSSL/1.1.0e libssh2/1.7.0_DEV
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 400 Bad Request
< Server: nps/2.3.1
< Mime-Version: 1.0
< Date: Wed, 25 Oct 2017 10:03:23 GMT
< Content-Type: text/html
< Content-Length: 4205
< X-Squid-Error: ERR_INVALID_URL 0
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from netentsec-nps-172.17.18.84
< Connection: close
<
{ [data not shown]
* Curl_http_done: called premature == 0
100 4205 100 4205 0 0 29450 0 --:--:-- --:--:-- --:--:-- 29822
* Closing connection 0
有意思的是吴趴，curl 連接上我的代理服務器后释涛，使用http協(xié)議去訪問sftp://127.0.0.1. 所以我們會得到Bad request (400)的響應涝婉。
如果我把127.0.0..1從代理中排除(e.g. export no_proxy=127.0.0.1)則請求就可以成功了哥力。
挖掘真相
為什么在連接代理后，curl會使用http協(xié)議去訪問sftp服務器呢？ 我在curl的repository中創(chuàng)建了issue. 從官方的回答中得到了答案吩跋。因為我正在使用老版本的curl(7.52), 這個版本不支持自動轉換隧道協(xié)議(tunneling). 在7.55.0版本后才支持這種自動轉換寞射。摘自issue的回復
Since 7.55.0, curl will enable tunneling automatically when you try to use SFTP over an HTTP proxy.
所以如果使用最新的curl, 這個問題應該就能解決了。接下來嘗試下載和編譯了最新的curl(7.56), 但是不幸的是仍然不能成功锌钮，并且返回了另一個錯誤:
* STATE: INIT => CONNECT handle 0x1a42ec8; line 1425 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* Trying 172.17.18.84...
* TCP_NODELAY set
* STATE: CONNECT => WAITCONNECT handle 0x1a42ec8; line 1477 (connection #0)
* Connected to 172.17.18.84 (172.17.18.84) port 8080 (#0)
* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x1a42ec8; line 1594 (connection #0)
* Marked for [keep alive]: HTTP default
* allocate connect buffer!
* Establish HTTP proxy tunnel to 127.0.0.1:22
* Server auth using Basic with user 'sftptest'
> CONNECT 127.0.0.1:22 HTTP/1.1
> Host: 127.0.0.1:22
> User-Agent: curl/7.57.0-DEV
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 503 Service Unavailable
< Server: nps/2.3.1
< Mime-Version: 1.0
< Date: Thu, 26 Oct 2017 07:09:25 GMT
< Content-Type: text/html
< Content-Length: 4069
< X-Squid-Error: ERR_CONNECT_FAIL 111
< Vary: Accept-Language
< Content-Language: en
注意到新的錯誤是503 Service Unavailable. 為什么會這樣呢桥温？我們可以一步一步分析原因。
首先轧粟，可以輕易發(fā)現(xiàn)連接代理服務器是成功的:
* Trying 172.17.18.84...
* TCP_NODELAY set
* STATE: CONNECT => WAITCONNECT handle 0x1a42ec8; line 1477 (connection #0)
* Connected to 172.17.18.84 (172.17.18.84) port 8080 (#0)
* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x1a42ec8; line 1594 (connection #0)
* Marked for [keep alive]: HTTP default
* allocate connect buffer!
* Establish HTTP proxy tunnel to 127.0.0.1:22
Tcp連接的狀態(tài)變化為: CONNECT => WAITCONNECT => WAITPROXYCONNECT => Establish HTTP proxy tunnel. 這個狀態(tài)變化顯然是正確的策治，代表連接代理服務器成功脓魏。
接下來嘗試連接sftp服務器:
* Server auth using Basic with user 'sftptest'
> CONNECT 127.0.0.1:22 HTTP/1.1
> Host: 127.0.0.1:22
> User-Agent: curl/7.57.0-DEV
> Proxy-Connection: Keep-Alive
但是得到的響應卻是:
< HTTP/1.1 503 Service Unavailable
也就是說兰吟，從代理服務器發(fā)起sftp連接到sftp服務器失敗 - 并且是sftp服務器服務不可用。這下真相逐漸浮出水面: 因為代理服務器嘗試去連接127.0.0.1:22, 但是這個地址對于代理服務器而言茂翔，是連接自己的22端口混蔼，而不是去連接sftp服務器！如果把這里的127.0.0.1替換為sftp服務器的內網地址珊燎，則可以成功:
* STATE: INIT => CONNECT handle 0x2238ec8; line 1425 (connection #-5000)
* Rebuilt URL to: sftp://sftptest@172.26.9.94/
* Added connection 0. The cache now contains 1 members
* Trying 172.17.18.84...
* TCP_NODELAY set
* STATE: CONNECT => WAITCONNECT handle 0x2238ec8; line 1477 (connection #0)
* Connected to 172.17.18.84 (172.17.18.84) port 8080 (#0)
* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x2238ec8; line 1594 (connection #0)
* Marked for [keep alive]: HTTP default
* allocate connect buffer!
* Establish HTTP proxy tunnel to 172.26.9.94:22
* Server auth using Basic with user 'sftptest'
> CONNECT 172.26.9.94:22 HTTP/1.1
> Host: 172.26.9.94:22
> User-Agent: curl/7.57.0-DEV
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* STATE: WAITPROXYCONNECT => SENDPROTOCONNECT handle 0x2238ec8; line 1573 (connection #0)
* CONNECT phase completed!
* SFTP 0x223f170 state change from SSH_STOP to SSH_INIT
* SFTP 0x223f170 state change from SSH_INIT to SSH_S_STARTUP
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x2238ec8; line 1608 (connection #0)
* SFTP 0x223f170 state change from SSH_S_STARTUP to SSH_HOSTKEY
* SSH MD5 fingerprint: 4cf03683e054a3398c91d76a16715e6b
* SSH host check: 2, key: <none>
* SFTP 0x223f170 state change from SSH_HOSTKEY to SSH_SESSION_FREE
* Marked for [closure]: SSH session free
* SFTP 0x223f170 state change from SSH_SESSION_FREE to SSH_STOP
* multi_done
* SSH DISCONNECT starts now
* SSH DISCONNECT is done
* Closing connection 0
* The cache now contains 0 members
結論
要解決curl使用sftp訪問127.0.0.1或localhost的問題惭嚣，可以采用:
- 方法1 （推薦）: 把127.0.0.1或localhost加入
no_proxy. 這樣無論使用哪個版本的curl, 都能正常工作。 - 方法2: 目標地址使用內網ip地址悔政，而不要使用127.0.0.1, 并且保證curl的版本在7.55以上晚吞。 但是這樣做的話，curl會先連接代理服務器谋国，再發(fā)起sftp請求槽地，性能會有損耗。
