今天對于數(shù)據(jù)收集的接口修改成nginx+lua+redis收集數(shù)據(jù)，cron腳本處理數(shù)據(jù)交胚。
環(huán)境
使用OpenResty + redis
2核4G centos6.4
nginx配置
server {
listen 80;
server_name t.com;
rewrite ^\/collect /LUA/ last;
rewrite ^(.*)$ /index.php?_RW_=$1 last;
location /LUA/{
default_type text/html;
content_by_lua_file /web/test/lua/collect.lua;
}
location / {
root /web/test/wwwroot;
}
}
collect.lua 請求處理腳本
local function breakpoint()
ngx.say('breakpoint');
ngx.eof();
end
local function getPostFormArgv()
local function explode(_str, seperator)
local pos, arr = 0, {}
for st, sp in function() return string.find(_str, seperator, pos, true) end do
table.insert(arr, string.sub(_str, pos, st - 1))
pos = sp + 1
end
table.insert(arr, string.sub(_str, pos))
return arr
end
local receive_headers = ngx.req.get_headers()
ngx.req.read_body()
local body = ngx.req.get_body_data()
local seperator = '--' .. string.sub(receive_headers["content-type"], 31)
local tmp = explode(body, seperator)
table.remove(tmp, 1)
table.remove(tmp)
local argv = {}
for k, v in pairs(tmp) do
local start_pos, end_pos, key, val = string.find(v, 'Content%-Disposition: form%-data; name="([^\"]+)"(.*)')
argv[key] = string.gsub(val, "[\r\n]", '')
end
return argv
end
local function getPostJsonArgv()
local json = require('cjson');
ngx.req.read_body()
local body = ngx.req.get_body_data()
if body == nil then
return {}
end
local argv = json.decode(body);
return argv
end
local function getPostArgv()
local receive_headers = ngx.req.get_headers()
local argv = {}
if receive_headers["content-type"] and string.sub(receive_headers["content-type"], 1, 20) == "multipart/form-data;" then
argv = getPostFormArgv()
else
argv = getPostJsonArgv()
end
return argv
end
local argv;
-- 獲取post參數(shù)
argv = getPostArgv();
if (argv.running == nil) then
ngx.say('{"code":101}');
return
end
-- 獲取header數(shù)據(jù)
local headers = ngx.req.get_headers();
argv.mac = headers.mac;
-- 存redis
local redis = require "resty.redis"
local red = redis:new()
red:set_timeout(1000) -- 1 sec
local ok, err = red:connect("127.0.0.1", 6380)
if not ok then
ngx.say("failed to connect: ", err)
return
end
local json = require('cjson');
local json_string = json.encode(argv);
red:lpush('ng_running', json_string);
red:close()
ngx.say('{"code":100}');
ngx.eof();
Nginx API for Lua https://github.com/openresty/lua-nginx-module#nginx-api-for-lua
壓力測試
先用Curl測試上面代碼是否存在問題
curl http://t.com/test -H "uid:1" -H "from:curl" -X POST -d'{"a":"1","b":"2"}'
使用ab測試
ab -c 500 -n 1000 -H "uid:1" -H "from:curl" -p post.text http://t.com/test
#運(yùn)行結(jié)果
Server Software: openresty/1.13.6.1
Server Hostname: 127.0.0.1
Server Port: 80
Document Path: /api/test
Document Length: 13 bytes
Concurrency Level: 500
Time taken for tests: 0.480 seconds
Complete requests: 2000
Failed requests: 0
Total transferred: 352000 bytes
Total body sent: 2094000
HTML transferred: 26000 bytes
Requests per second: 4166.27 [#/sec] (mean)
Time per request: 120.011 [ms] (mean)
Time per request: 0.240 [ms] (mean, across all concurrent requests)
Transfer rate: 716.08 [Kbytes/sec] received
4259.85 kb/s sent
4975.92 kb/s total
對比我用php寫的收集腳本际邻，同樣是接收數(shù)據(jù)寫入redis装哆，吞吐量在260左右罐脊，lua的方案在4200左右。
Server Software: openresty/1.13.6.1
Server Hostname: 127.0.0.1
Server Port: 80
Document Path: /api/test2
Document Length: 24 bytes
Concurrency Level: 500
Time taken for tests: 7.617 seconds
Complete requests: 2000
Failed requests: 0
Total transferred: 386000 bytes
Total body sent: 2104000
HTML transferred: 48000 bytes
Requests per second: 262.59 [#/sec] (mean)
Time per request: 1904.128 [ms] (mean)
Time per request: 3.808 [ms] (mean, across all concurrent requests)
Transfer rate: 49.49 [Kbytes/sec] received
269.77 kb/s sent
319.26 kb/s total
ab性能指標(biāo)
在進(jìn)行性能測試過程中有幾個(gè)指標(biāo)比較重要：
1蜕琴、吞吐率（Requests per second）
服務(wù)器并發(fā)處理能力的量化描述萍桌，單位是reqs/s，指的是在某個(gè)并發(fā)用戶數(shù)下單位時(shí)間內(nèi)處理的請求數(shù)凌简。某個(gè)并發(fā)用戶數(shù)下單位時(shí)間內(nèi)能處理的最大請求數(shù)上炎，稱之為最大吞吐率。
記壮А：吞吐率是基于并發(fā)用戶數(shù)的藕施。這句話代表了兩個(gè)含義：
a、吞吐率和并發(fā)用戶數(shù)相關(guān)
b凸郑、不同的并發(fā)用戶數(shù)下裳食，吞吐率一般是不同的
計(jì)算公式：總請求數(shù)/處理完成這些請求數(shù)所花費(fèi)的時(shí)間，即
Request per second=Complete requests/Time taken for tests
必須要說明的是芙沥，這個(gè)數(shù)值表示當(dāng)前機(jī)器的整體性能诲祸，值越大越好。
2而昨、并發(fā)連接數(shù)（The number of concurrent connections）
并發(fā)連接數(shù)指的是某個(gè)時(shí)刻服務(wù)器所接受的請求數(shù)目救氯，簡單的講，就是一個(gè)會(huì)話歌憨。
3着憨、并發(fā)用戶數(shù)（Concurrency Level）
要注意區(qū)分這個(gè)概念和并發(fā)連接數(shù)之間的區(qū)別，一個(gè)用戶可能同時(shí)會(huì)產(chǎn)生多個(gè)會(huì)話务嫡，也即連接數(shù)甲抖。在HTTP/1.1下，IE7支持兩個(gè)并發(fā)連接心铃，IE8支持6個(gè)并發(fā)連接准谚，F(xiàn)ireFox3支持4個(gè)并發(fā)連接，所以相應(yīng)的于个，我們的并發(fā)用戶數(shù)就得除以這個(gè)基數(shù)。
4暮顺、用戶平均請求等待時(shí)間（Time per request）
計(jì)算公式：處理完成所有請求數(shù)所花費(fèi)的時(shí)間/（總請求數(shù)/并發(fā)用戶數(shù)）厅篓，即：
Time per request=Time taken for tests/（Complete requests/Concurrency Level）
5秀存、服務(wù)器平均請求等待時(shí)間（Time per request:across all concurrent requests）
計(jì)算公式：處理完成所有請求數(shù)所花費(fèi)的時(shí)間/總請求數(shù)，即：
Time taken for/testsComplete requests
可以看到羽氮，它是吞吐率的倒數(shù)或链。
同時(shí)，它也等于用戶平均請求等待時(shí)間/并發(fā)用戶數(shù)档押，即
Time per request/Concurrency Level
ab報(bào)錯(cuò)處理
apr_socket_recv: Connection reset by peer (104)
解決方法：加參數(shù) -r
