2018年的最后幾天平臺(tái)迭代到了V6版本碳竟，代號(hào)Veronica（維羅妮卡）顽染。（Veronica名字取自電影復(fù)仇者聯(lián)盟2漾岳，電影中的Veronica是由布魯斯·班納（綠巨人）和托尼·斯塔克（鋼鐵俠）共同完成，Veronica系統(tǒng)包含多個(gè)功能模塊粉寞，反浩克裝甲是其中之一尼荆，反浩克裝甲的作用相當(dāng)于外勤特工，其余功能模塊起著支持唧垦、補(bǔ)給作用捅儒。除反浩克外其余已知的功能模塊包括：獨(dú)立衛(wèi)星通信，獨(dú)立操作，戰(zhàn)損支持（出現(xiàn)戰(zhàn)損可調(diào)用備件更換）巧还，監(jiān)獄式隔離倉(cāng)等鞭莽，有點(diǎn)跑題了....之所以取名Veronica ，是因?yàn)檫\(yùn)維平臺(tái)跟電影中有很多向像之處麸祷，首先平臺(tái)本身獨(dú)立工作澎怒，且并不具備任何運(yùn)維功能，全部依靠調(diào)用外部API接口外部應(yīng)用阶牍，同時(shí)平臺(tái)本身可以通過(guò)分布式部署喷面，使不同的節(jié)點(diǎn)一同來(lái)處理公共的任務(wù)，諸如：監(jiān)控作業(yè)走孽，項(xiàng)目發(fā)布惧辈，數(shù)據(jù)采集（采集這塊我用到了Salt，在Veronica 中磕瓷，Salt master也是多主架構(gòu)的部署方式盒齿，即由不同的Veronica 節(jié)點(diǎn)來(lái)進(jìn)行配置管理、信息采集困食、同步模塊等县昂，這個(gè)下期會(huì)說(shuō)到），統(tǒng)計(jì)陷舅，計(jì)算及一些耗時(shí)任務(wù)倒彰，比如創(chuàng)建虛擬機(jī)。

下文給大家一個(gè)思路莱睁，就是借助Celery分布式特性待讳，通過(guò)多節(jié)點(diǎn)運(yùn)行數(shù)據(jù)監(jiān)控，然后存儲(chǔ)到Redis仰剿，最后由前端Dashboard讀取數(shù)據(jù)创淡。先上一張效果圖：

image.png

image.png

一、 環(huán)境準(zhǔn)備

1.Celery 3.1.26南吮，官網(wǎng)http://docs.celeryproject.org/en/latest/
2.Celery-with-redis 3.0
3.QUEUE及CELERY_RESULT_BACKEND 采用了REDIS 3.2.0 (后面會(huì)換Rabbitmq琳彩，官方潛力推介)
4.Django 1.6.6
5.Django-celery (3.1.17)

二、食用方法

1.Celery：

我的環(huán)境是3臺(tái)服務(wù)器部凑，一臺(tái)主控露乏，兩臺(tái)代理（處理任務(wù)），每臺(tái)代理跑4個(gè)隊(duì)列（如圖1中的25-1---25-4及26-1---26-4）
代理25服務(wù)器的啟動(dòng)命令 python manage.py celery worker -n 25 -l info -Q 25-1,25-2,25-3,25-4
代理26服務(wù)器的啟動(dòng)命令 python manage.py celery worker -n 26 -l info -Q 26-1,26-2,26-3,26-4
主控24服務(wù)器的啟動(dòng)命令python manage.py celery beat -n 24 -l info 及 python manage.py celery worker -n 24 -l info -Q default

代碼結(jié)構(gòu)如下：

image.png

然后3臺(tái)分別在settings.py中注冊(cè)定時(shí)任務(wù)模塊：

CELERY_IMPORTS = ('cmdb.mycelery.tasks.vm_task',
                  'cmdb.mycelery.schedule.cmdb_cron.cmdb_f',
                  'cmdb.mycelery.schedule.openfalconagent_cron.openfalcon_f',
                  'cmdb.mycelery.schedule.salt_cron.salt_f',
                  'cmdb.mycelery.schedule.dashboard_cron.openfalcon_f',
)

2.Djcelery：

使用了djcelery涂邀，可以在任務(wù)中方便的直接操作Django數(shù)據(jù)庫(kù)瘟仿，而且最終的任務(wù)可以在Django的后臺(tái)中查看和修改相關(guān)的任務(wù)。但是我們需要通過(guò)運(yùn)維平臺(tái)來(lái)調(diào)用djcelery的models來(lái)進(jìn)行作業(yè)的增比勉、刪劳较、改驹止，而不是讓運(yùn)維小伙伴通過(guò)ADMIN后臺(tái)去修改。

后臺(tái)代碼：

@csrf_exempt
@login_required
def schedule_interval_ajax(request):
    """獲取interval观蜗，組成select"""
    interval_dict = {}
    intervalschedule_list = celery_models.IntervalSchedule.objects.all()
    for i in intervalschedule_list:
        interval_dict[i.id] = str(i.every) + ' ' + str(i.period)
    return HttpResponse(simplejson.dumps(interval_dict, ensure_ascii=False))


@csrf_exempt
@login_required
def schedule_crontab_ajax(request):
    """獲取crontab臊恋，組成select"""
    crontab_dict = {}
    crontabschedule_list = celery_models.CrontabSchedule.objects.all()
    for i in crontabschedule_list:
        crontab_dict[i.id] = str(i.minute) + ' ' + str(i.hour) + ' ' + str(i.day_of_week) + ' ' + str(i.day_of_month) + ' ' + str(i.month_of_year)
    return HttpResponse(simplejson.dumps(crontab_dict, ensure_ascii=False))

@csrf_exempt
@login_required
@my_permissionVerify
def schedule_edit_ajax(request):
    """修改作業(yè)"""
    if request.method == 'POST':
        schedule_type = request.POST['schedule_type']
        try:
            schedule_crontab_id = request.POST['schedule_crontab']
        except:
            schedule_crontab_id = ''
        try:
            schedule_interval_id = request.POST['schedule_interval']
        except:
            schedule_interval_id = ''
        if request.POST['schedule_enabled'] == 'true':
            schedule_enabled_id = 1
        else:
            schedule_enabled_id = 0
        schedule_args = request.POST['schedule_args']
        schedule_kwargs = request.POST['schedule_kwargs']
        schedule_name = request.POST['schedule_name']
        schedule_queue = request.POST['schedule_queue']
        try:
            celery_models.PeriodicTask.objects.filter(name=schedule_name).update(interval = schedule_interval_id,
                                                                                 name = schedule_name,
                                                                                 crontab = schedule_crontab_id,
                                                                                 enabled = schedule_enabled_id,
                                                                                 args = schedule_args,
                                                                                 kwargs = schedule_kwargs,
                                                                                 queue = schedule_queue)
            """保存"""
            celery_models.PeriodicTask.objects.get(name=schedule_name).save()
            return HttpResponse('{"status":1}')
        except Exception as err:
            print err
            return HttpResponse('{"status":0, "err":%s}'% err)

    else:
        name = request.GET['schedule_name']
        periodictask_dict = {}
        interval_dict = {}
        crontab_dict = {}
        periodictask_info = celery_models.PeriodicTask.objects.filter(name=name)
        for i in periodictask_info:
            periodictask_dict['name'] = i.name
            periodictask_dict['task'] = i.task
            interval_dict['id'] = i.interval_id
            interval_dict['context'] = str(i.interval)
            crontab_dict['id'] = i.crontab_id
            crontab_dict['context'] = str(i.crontab)
            periodictask_dict['interval_dict'] = interval_dict
            periodictask_dict['crontab_dict'] = crontab_dict
            periodictask_dict['queue'] = i.queue
            periodictask_dict['args'] = i.args
            periodictask_dict['kwargs'] = i.kwargs
            periodictask_dict['enabled'] = i.enabled
        return HttpResponse(simplejson.dumps(periodictask_dict, ensure_ascii=False))

JS代碼：

function schedule_type_hidden (obj){
    if (obj.value=="1"){
        document.getElementById("add_schedule_interval").style.display ="block";
        document.getElementById("add_schedule_crontab").style.display ="none";
    }else{
        document.getElementById("add_schedule_interval").style.display ="none";
        document.getElementById("add_schedule_crontab").style.display ="block";
        }
}

$(function(){
    $("#schedule_interval_select").focus(function(){
    $.ajax({
        async: false,
        type: "POST",
        url : "../schedule_interval_ajax/",
        data : $("#submit_form").serialize(),
        cache: false,
        dataType: "json",
        beforeSend:function(){
            Metronic.blockUI({animate: true});
        },
        success: function(obj) {
            for (var id in obj){
            $("#schedule_interval_select").append("<option value="+id+">"+obj[id]+"</option>");
                }
            $("#schedule_interval_select").unbind("focus");
                },
        complete: function() {
            Metronic.unblockUI();
        },
            });
            return false;
            });
        });

$(function(){
    $("#schedule_crontab_select").focus(function(){
    $.ajax({
        async: false,
        type: "POST",
        url : "../schedule_crontab_ajax/",
        data : $("#submit_form").serialize(),
        cache: false,
        dataType: "json",
        beforeSend:function(){
            Metronic.blockUI({animate: true});
        },
        success: function(obj) {
            for (var id in obj){
            $("#schedule_crontab_select").append("<option value="+id+">"+obj[id]+"</option>");
                }
            $("#schedule_crontab_select").unbind("focus");
                },
        complete: function() {
            Metronic.unblockUI();
        },
            });
            return false;
            });
        });


$(function(){
    $("#edit_schedule_interval_select").focus(function(){
    $.ajax({
        async: false,
        type: "POST",
        url : "../schedule_interval_ajax/",
        data : $("#submit_form").serialize(),
        cache: false,
        dataType: "json",
        beforeSend:function(){
            Metronic.blockUI({animate: true});
        },
        success: function(obj) {
            $("#edit_schedule_interval_select").empty()
            for (var id in obj){
            $("#edit_schedule_interval_select").append("<option value="+id+">"+obj[id]+"</option>");
                }
            $("#edit_schedule_interval_select").unbind("focus");
                },
        complete: function() {
            Metronic.unblockUI();
        },
            });
            return false;
            });
        });

$(function(){
    $("#edit_schedule_crontab_select").focus(function(){
    $.ajax({
        async: false,
        type: "POST",
        url : "../schedule_crontab_ajax/",
        data : $("#submit_form").serialize(),
        cache: false,
        dataType: "json",
        beforeSend:function(){
            Metronic.blockUI({animate: true});
        },
        success: function(obj) {
            $("#edit_schedule_crontab_select").empty()
            for (var id in obj){
            $("#edit_schedule_crontab_select").append("<option value="+id+">"+obj[id]+"</option>");
                }
            $("#edit_schedule_crontab_select").unbind("focus");
                },
        complete: function() {
            Metronic.unblockUI();
        },
            });
            return false;
            });
        });



function schedule_edit(){
    $(".fa-edit").bind('click', function() {
        var schedule_name =$(this).parent().parent().find('td').eq(0).attr("id");
        $.ajax({
            type: "GET",
            url: "../schedule_edit_ajax/?schedule_name="+schedule_name,
            cache: false,
            async:true,
            dataType: "json",
            beforeSend:function(){
                Metronic.blockUI({animate: true});
            },
            complete: function() {
                Metronic.unblockUI();
            },
            success: function(obj)  {
                $('#edit_schedule_name').val(obj['name'])
                $('#edit_schedule_queue').val(obj['queue'])
                $('#edit_schedule_args').val(obj['interval_id'])
                $('#edit_schedule_args').val(obj['args'])
                $('#edit_schedule_kwargs').val(obj['kwargs'])
                if (obj['interval_dict']['id'] == null){
                  $('#edit_schedule_type').val("2")
                    document.getElementById("edit_add_schedule_interval").style.display ="none";
                    document.getElementById("edit_add_schedule_crontab").style.display ="block";
                    $("#edit_schedule_crontab_select").append("<option value="+obj['crontab_dict']['id']+">"+obj['crontab_dict']['context']+"</option>");
                }
                if (obj['crontab_dict']['id'] == null){
                  $('#edit_schedule_type').val("1")
                    document.getElementById("edit_add_schedule_interval").style.display ="block";
                    document.getElementById("edit_add_schedule_crontab").style.display ="none";
                    $("#edit_schedule_interval_select").append("<option value="+obj['interval_dict']['id']+">"+obj['interval_dict']['context']+"</option>");
                }
                if (obj['enabled'] == true ){
                    $('#edit_schedule_enabled').bootstrapSwitch('toggleState');
                    $('#edit_schedule_enabled').bootstrapSwitch('state', true);
                    $("#edit_schedule_enableds").val('true')
                }else{
                    $('#edit_schedule_enabled').bootstrapSwitch('toggleState');
                    $('#edit_schedule_enabled').bootstrapSwitch('state', false);
                    $("#edit_schedule_enableds").val('false')
                }
                }
                });
        $("#edit_schedule_enabled").on('switchChange.bootstrapSwitch', function(e, status) {
            if (status == false){
            $("#edit_schedule_enableds").val('false');
            }else{
            $("#edit_schedule_enableds").val('true');
            }
        });
    $('#responsive2 form').submit(function(){
        $.ajax({
            type: "POST",
            data: $('#responsive2 form').serialize(),
            url: "../schedule_edit_ajax/",
            cache: false,
            async:true,
            dataType: "json",
            beforeSend:function(){
                Metronic.blockUI({animate: true});
            },
            complete: function() {
                Metronic.unblockUI();
            },
            success: function(obj)  {
                  if (obj['status'] == "1"){
                      alert('定時(shí)調(diào)度修改成功');
                      $('#responsive2').modal('hide');
                      $('#responsive2 form')[0].reset();
                  }else {
                      alert('定時(shí)調(diào)度修改失敗,錯(cuò)誤為:'+obj['err']);
                   }
                }
                });
                return false;
                });
                });
           $('#responsive2').on('hide.bs.modal', function () {
              location.reload();
            });
            }
            $(document).ready(function(){
                schedule_edit();
                })

3.最后：

后面要做的就是通過(guò)異步的任務(wù)隊(duì)列對(duì)獲取你想要的數(shù)據(jù)，圖上我的作業(yè)是通過(guò)OPENFALCON的接口獲取負(fù)載排名墓捻、IO排名捞镰，進(jìn)出流量排名等。然后存儲(chǔ)到Redis毙替，在由Dashboard通過(guò)局部刷新分別從Redis中讀取數(shù)據(jù)岸售。這樣做的好處，不會(huì)因?yàn)樵L問(wèn)的客戶端多厂画，導(dǎo)致頻繁發(fā)起對(duì)各個(gè)監(jiān)控系統(tǒng)的請(qǐng)求凸丸。同時(shí)可以將監(jiān)控作業(yè)分散在若干機(jī)器內(nèi)，實(shí)現(xiàn)橫向的擴(kuò)展（絕大部分商業(yè)監(jiān)控的辦法）袱院。

獲取報(bào)警作業(yè)屎慢，代碼如下：

r = redis_conect_db5()

@task()
def triggers_list():
    cursor.execute(sql)
    triggers =cursor.fetchall()
    triggers_list=[]
    for i in triggers:
        triggers_dict={}
        triggers_dict['host'] = i[0]
        triggers_dict['description'] = i[1]
        triggers_dict['lastchange'] = Calculate_date_today(str(i[2]))
        triggers_dict['priority'] = i[3]
        triggers_dict['metric'] = ''.join(i[4].split('/')[1:]).replace(',','<br>')
        triggers_dict['cond'] = i[5]
        triggers_list.append(triggers_dict)
    cursor.close()
    conn.close()
    try:
        r.set("dashboard:triggers_list", json.dumps(triggers_list))
    except Exception as err:
        print err
    return triggers_list

前臺(tái)局部刷新獲取數(shù)據(jù)，代碼如下：

r = redis_conect_db5()

def triggers_list():
    try:
        print 'triggers from redis'
        triggers_list = json.loads(r.get("dashboard:triggers_list"))
    except:
        triggers_list = []
    return triggers_list