一盖高、簡(jiǎn)介
Bearded-avenger(cifv3)是一個(gè)定期爬取公開威脅數(shù)據(jù)的平臺(tái)（the fastest way to consume threat intelligence）凯亮。通過(guò)定制yaml規(guī)則對(duì)定期公布威脅數(shù)據(jù)的站點(diǎn)進(jìn)行數(shù)據(jù)爬取，并歸納整理出統(tǒng)一的格式方便安全人員使用芹橡。
數(shù)據(jù)庫(kù)有兩種選擇：Sqlite和Elasticsearch细燎，我選擇了強(qiáng)大的Elasticsearch
github：https://github.com/csirtgadgets/bearded-avenger
二坎怪、準(zhǔn)備一個(gè)搭建cifv3的服務(wù)器環(huán)境
建議使用：（本人本次搭建在ESX上）
操作系統(tǒng)：ubuntu14.04
內(nèi)核數(shù)：8個(gè)
內(nèi)存：16G
三骨饿、選擇一個(gè)最新的穩(wěn)定版本下載到本地
本次使用：https://github.com/csirtgadgets/bearded-avenger/releases/tag/3.0.0a16
cd /root/
wget https://github.com/csirtgadgets/bearded-avenger/archive/3.0.0a16.tar.gz
四、前期網(wǎng)絡(luò)環(huán)境及下載源的設(shè)置
1. 更改為阿里云源
mv /etc/apt/sources.list /etc/apt/sources.list.bak
vim /etc/apt/sources.list
deb http://mirrors.aliyun.com/ubuntu/ trusty main multiverse restricted universe
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main multiverse restricted universe
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main multiverse restricted universe
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main multiverse restricted universe
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main multiverse restricted universe
deb-src http://mirrors.aliyun.com/ubuntu/ trusty main multiverse restricted universe
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main multiverse restricted universe
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main multiverse restricted universe
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main multiverse restricted universe
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main multiverse restricted universe
2. ubuntu配置pip國(guó)內(nèi)鏡像源
cd ~
mkdir .pip
cd .pip
vim pip.conf
[global]
trusted-host = mirrors.ustc.edu.cn
index-url = https://mirrors.ustc.edu.cn/pypi/web/simple
3. 更新系統(tǒng)源和軟件
apt-get update
apt-get upgrade
五顾翼、一些可能出錯(cuò)的提前避免
由于使用的是一鍵安裝腳本投放，所以每次出錯(cuò)都要重新執(zhí)行一遍，而且其中可能命令重復(fù)又會(huì)出現(xiàn)新的問(wèn)題适贸，在此給出了所有可能出錯(cuò)的解決方法灸芳，可以運(yùn)行解決辦法中的命令涝桅，讓大家講問(wèn)題扼殺在萌芽期。
1. ImportError: No module named packaging.version
錯(cuò)誤截圖：
pip.jpg
解決辦法：
apt-get purge -y python-pip
wget https://bootstrap.pypa.io/get-pip.py
python ./get-pip.py
apt-get install python-pip
2. geoipupdate失斃友（國(guó)內(nèi)更新geoip庫(kù)可能較慢冯遂，超過(guò)超時(shí)時(shí)間）
錯(cuò)誤截圖：
geoip.jpg
解決辦法：
add-apt-repository ppa:maxmind/ppa
aptitude update
aptitude install geoipupdate
geoipupdate -v
3. ansible_env錯(cuò)誤
錯(cuò)誤截圖：
ansible.jpg
解決辦法：
vim bearded-avenger/deployment/ubuntu14/roles/ubuntu14/tasks/user.yml
ansible_env.SUDO_USER 改成 ansible_env.USER
ansible2.png
4. 執(zhí)行測(cè)試腳本時(shí)出錯(cuò)（存在國(guó)內(nèi)無(wú)法訪問(wèn)的域名）
錯(cuò)誤截圖：
test.png
部分出錯(cuò)代碼：
"=================================== FAILURES ===================================",
"______________________________ test_gatherer_asn _______________________________",
"",
" def test_gatherer_asn():",
" a = Asn(fast=False)",
" ",
" def _resolve(i):",
" return data",
" ",
" a._resolve_ns = _resolve",
" x = a.process(Indicator(indicator='216.90.108.0'))",
" ",
"> assert x.asn == '23028'",
"E assert None == '23028'",
"E + where None = {\\n \"indicator\": \"216.90.108.0\",\\n \"itype\": \"ipv4\"\\n}.asn",
"",
"test/test_gatherer_asn.py:28: AssertionError",
"----------------------------- Captured stderr call -----------------------------",
"2017-02-17 07:08:11,757 - INFO - cif.utils[22][MainThread] - \u001b[32m0.108.90.216.origin.asn.cymru.com - The DNS operation timed out after 5.0050868988 seconds -- this may be normal\u001b[0m",
"2017-02-17 07:08:11,757 - INFO - cif.utils[22][MainThread] - \u001b[32m\u001b[32m0.108.90.216.origin.asn.cymru.com - The DNS operation timed out after 5.0050868988 seconds -- this may be normal\u001b[0m\u001b[0m",
"2017-02-17 07:08:11,757 - INFO - cif.utils[22][MainThread] - \u001b[32m\u001b[32m\u001b[32m0.108.90.216.origin.asn.cymru.com - The DNS operation timed out after 5.0050868988 seconds -- this may be normal\u001b[0m\u001b[0m\u001b[0m",
"================ 1 failed, 17 passed, 9 skipped in 7.83 seconds ================"
解決辦法：
暫時(shí)刪掉過(guò)不去的測(cè)試腳本
cd bearded-avenger/test
mv test_gatherer_asn.py test_gatherer_asn.py.bak
5. docker pull elasticsearch鏡像時(shí)可能超時(shí)
在本地安裝docker并pull elasticsearch:2
流程：
curl -sSL http://acs-public-mirror.oss-cn-hangzhou.aliyuncs.com/docker-engine/internet | sh -
docker pull elasticsearch:2
六、執(zhí)行一件安裝腳本：
cd bearded-avenger/deployment/ubuntu14
bash bootstrap_elasticsearch.sh
成功安裝的截圖：
chengong.png
七谒获、嘗試爬取黑IP黑域名等數(shù)據(jù)到elasticsearch
su - cif
csirtg-smrt --client cif --fireball -r /etc/cif/rules/default/csirtg.yml -f port-scanners -d
成功執(zhí)行的截圖：
feed.png
八蛤肌、查看elasticsearch數(shù)據(jù)
目前elasticsearch還沒(méi)有安裝head、kopf等插件批狱，只能用curl命令
可拷貝其他elasticsearch中的plugins目錄到本地的指定目錄裸准。
root@elk:~/bearded-avenger/test# find / -name elasticsearch.in.sh
/var/lib/docker/aufs/diff/b8d77a9eee69729f60d454e22b262abd93ebd135fedf92faff4b7e5d950a1194/usr/share/elasticsearch/bin/elasticsearch.in.sh
/var/lib/docker/aufs/mnt/1b866f98ba81754fdee761eb64bb3782594d3ac4890ffa271d9f5c6abc929594/usr/share/elasticsearch/bin/elasticsearch.in.sh
root@elk:~/bearded-avenger/test# cd /var/lib/docker/aufs/mnt/1b866f98ba81754fdee761eb64bb3782594d3ac4890ffa271d9f5c6abc929594/usr/share/elasticsearch/
root@elk:/var/lib/docker/aufs/mnt/1b866f98ba81754fdee761eb64bb3782594d3ac4890ffa271d9f5c6abc929594/usr/share/elasticsearch# ll
total 52
drwxr-xr-x 13 root root 4096 Feb 17 08:05 ./
drwxr-xr-x 72 root root 4096 Feb 17 07:05 ../
drwxr-xr-x 2 root root 4096 Feb 17 08:05 bin/
drwxr-xr-x 3 bind ssh 4096 Feb 7 15:54 config/
drwxr-xr-x 2 bind ssh 4096 Feb 7 15:53 data/
drwxr-xr-x 2 root root 4096 Feb 7 15:53 lib/
drwxr-xr-x 2 bind ssh 4096 Feb 7 15:53 logs/
drwxr-xr-x 5 root root 4096 Feb 7 15:53 modules/
-rw-r--r-- 1 root root 150 Jan 3 06:51 NOTICE.txt
drwxr-xr-x 4 bind ssh 4096 Feb 17 08:02 plugins/
-rw-r--r-- 1 root root 8700 Jan 3 06:51 README.textile
root@elk:/var/lib/docker/aufs/mnt/1b866f98ba81754fdee761eb64bb3782594d3ac4890ffa271d9f5c6abc929594/usr/share/elasticsearch# cd plugins/
root@elk:/var/lib/docker/aufs/mnt/1b866f98ba81754fdee761eb64bb3782594d3ac4890ffa271d9f5c6abc929594/usr/share/elasticsearch/plugins# ll
total 16
drwxr-xr-x 4 bind ssh 4096 Feb 17 08:02 ./
drwxr-xr-x 13 root root 4096 Feb 17 08:05 ../
drwxr-xr-x 5 bind ssh 4096 Feb 17 07:59 head/
drwxr-xr-x 8 bind ssh 4096 Feb 17 07:59 kopf/
九、其他配置
修改elasticsearch.in.sh 文件中的配置信息赔硫，保證發(fā)揮elasticsearch的最大特性
將此處兩個(gè)值設(shè)置相同切小于等于最大內(nèi)存數(shù)的1/2炒俱，此處可設(shè)置成8g
其他參數(shù)根據(jù)自己需求進(jìn)行修改
es.png
