Redis是一個(gè)高效的內(nèi)存數(shù)據(jù)庫(kù)叛氨， 根據(jù)多方面了解漏隐，
我們計(jì)劃選用Redis數(shù)據(jù)庫(kù)作為CSMD3.0的數(shù)據(jù)存儲(chǔ)方案
歡迎加入QQ群:<a >CTP/易盛(行情交易)開發(fā)</a>
測(cè)試環(huán)境搭建:
centos 安裝redis:
#yum install redis
啟動(dòng)redis服務(wù):
#service redis start
使用Redis客戶端測(cè)試Redis服務(wù)器工作是否正常：
#redis-cli
127.0.0.1:6379> set string1 "CCCCCCCCCC"
OK
127.0.0.1:6379> get string1
"CCCCCCCCCC"
127.0.0.1:6379>
把字符串"CCCCCCCCCC" 寫入 string1, 形成 string1 ==> "CCCCCCCCCC" 的鍵值對(duì)(key-value),
可以判斷Redis服務(wù)器工作正常计维。
安裝官方C++編程接口hiredis:
yum install hiredis-devel.x86_64
在線保存CTP數(shù)據(jù)立由，或者離線存儲(chǔ)深度行情數(shù)據(jù)到Redis:
//寫數(shù)據(jù)到Redis
int insert_to_redis(char* key, LXTick* value, CThostFtdcDepthMarketDataField* pDepthMarketData)
{
char score[128];
snprintf(score, sizeof(score)-1, "%ld", value->ticktimestamp);
redisReply* reply = (redisReply*)redisCommand(redis, "zadd %s %s %b", key, score, value, sizeof(LXTick));
freeReplyObject(reply);
return 0;
}
我們使用python程序從Cassandra數(shù)據(jù)庫(kù)取出兩年數(shù)據(jù)存入Redis數(shù)據(jù)庫(kù)给赞，用于測(cè)試舀武。
安裝apache2:
yum install httpd
安裝php5:
yum install php5
安裝phpredis:
yum install php5-redis
首先測(cè)試php是否功能正常：
#vi info.php
<?php
phpinfo();
?>
然后在shell執(zhí)行: php info.php 看輸出是否正常, 如果正常代表php已經(jīng)工作。
接下來(lái)把 info.php 復(fù)制到 /var/www/html/
然后瀏覽器打開 http://192.168.1.207:8080/info.php
如果正常顯示php和apache信息代表 apache與php均工作正常骏啰。
列出所有key getkeys.php (http://192.168.1.207:8080/getkeys.php?key=2015&db=2):
<?php
$redis = new Redis();
$redis->connect('192.168.1.207', 6379);
$redis->select($_GET["db"]);
$ticks = $redis->keys($_GET["key"]);
foreach($ticks as $i => $tick)
{
echo "<a href="getdata1.php?cc=$tick">$tick</a> ";
echo $redis->zcount($tick, 0, 99999999999999999); //顯示所有key
echo "<br />";
}
?>
點(diǎn)擊某合約獲取數(shù)據(jù) getdata1.php (http://192.168.1.207:8080/getdata1.php?cc=CZCE:RM:RM509:20150402):
<?php
$redis = new Redis();
$redis->connect('192.168.1.207', 6379);
$redis->select(2);
$ticks = $redis->zrange($_GET["cc"], 0, -1);
foreach($ticks as $i => $tick)
{
echo $tick;
}
?>
使用python測(cè)試獲取一個(gè)合約一個(gè)月內(nèi)所有Tick數(shù)據(jù),
以下測(cè)試結(jié)果的前一項(xiàng)是nginx+文件, 后一項(xiàng)是 apache + redis
nginx和apache2均啟用gzip壓縮的測(cè)試結(jié)果:
總文件大小 耗時(shí) 速率(解壓后文件體積/時(shí)間)
49093660kB, 0.550s 679.773Mbps (win2008 + nginx + 文件)
97384056kB, 1.563s 475.355Mbps (centos + apache + phpredis + redis)
均不啟用gzip壓縮的測(cè)試結(jié)果:
49093660KB, 4.1779999733s 89.6493306647Mbps (win2008 + nginx + 文件)
97384056kB, 8.61899995804s 86.2027367322Mbps (centos + apache + phpredis + redis)
由上表可見, 在不啟用壓縮條件下php + redis速率基本與nginx+文件方案相當(dāng)节吮，
附python測(cè)試腳本:
#--encoding=utf-8--
import requests
from Queue import Queue
from time import time
from threading import Thread
class HttpThread(Thread):
def __init__(self,url,queue,session):
Thread.__init__(self)
self.url=url
self.queue=queue
self.session=session
def run(self):
rep=self.session.get(self.url)
self.queue.put(rep.content)
def multithread_download(baseurl,batchparams,session):
st=time()
threads = []
downloadq=Queue()
for p in batchparams:
t=HttpThread(baseurl %p,downloadq,session)
t.start()
threads.append(t)
for t in threads:
t.join()
size=0
while not downloadq.empty():
t=downloadq.get()
size+=len(t)
tt=time()-st
print 'Http download data,Total size:%sKB,total time:%s '%(size,tt),'speed:%sMbps' %str(size8/(10241024*tt))
def test_http_download():
s=requests.Session()
s.get("http://192.168.1.206:8081/")
# s.headers['Accept-Encoding']='gzip,deflate'
multithread_download("http://192.168.1.206:8081/CTP/CFFEX/IF/IF1606/2016-05-%s/IF1606-P-tick-2016-05-%s.dat",
[(i,i) for i in ['03','04','05','06','09','10','11','12','13','16','17','18','19','20','23','24','25','26','27','30','31']], s)
def test_redishttp_download():
from cStringIO import StringIO
from lxml import etree
from lxml.cssselect import CSSSelector
s=requests.Session()
s.get("http://192.168.1.207:8080/")
# s.headers['Accept-Encoding']='deflate'
>rep=requests.get("http://192.168.1.207:8080/getkeys.php?key=CFFEX:IF:IF1310*&db=2").content
tree=etree.parse(StringIO(rep), etree.HTMLParser())
links=[e.attrib["href"] for e in CSSSelector("a")(tree)]
multithread_download("http://192.168.1.207:8080/%s", links, s)
test_http_download()
test_redishttp_download()
