背景####
最近上線的部分業(yè)務(wù)虛擬機(jī)存在定時同步集群文件呕诉，造成同步階段網(wǎng)絡(luò)流量飆高影響改宿主機(jī)上其他業(yè)務(wù)虛擬機(jī)墨吓。由于計(jì)算節(jié)點(diǎn)采用的兩塊1000M網(wǎng)卡做的Bond1，理論最大傳輸也就120MB/s左右。如果虛擬機(jī)出現(xiàn)網(wǎng)絡(luò)搶占的問題释簿，就不得不對其限速。
虛擬機(jī)
宿主機(jī)
從監(jiān)控可以看到硼莽，有一臺虛擬機(jī)每一個小時便會有有次入口流量高峰庶溶，幾乎跑慢網(wǎng)卡流量，同時這臺虛擬機(jī)磁盤WriteOPS開始增高懂鸵，說明這段時間虛擬機(jī)上業(yè)務(wù)傳輸數(shù)據(jù)已經(jīng)落盤偏螺。同時每隔一小時還有磁盤Read OPS高峰〈夜猓看來限速必不可免啊套像。
操作####
可以從三個方面著手網(wǎng)卡的流量限速。
OVS隊(duì)列+流表
libvirtd限速接口
Neutron QosPolicy
備注：本文主要采用第1種方式實(shí)現(xiàn)限速终息，簡單涉及第2夺巩，3種。
OVS隊(duì)列+流表
openvswitch的Port Qos policy只支持HTB
原理如下：
- 在虛擬機(jī)port上創(chuàng)建一條QOS
- 一條QOS隊(duì)列對應(yīng)一條Queue周崭，可以是對應(yīng)多條Queue
- 規(guī)則OVS通過流表
通過ovs-vsctl show查到虛擬機(jī)接到ovs上的tap網(wǎng)卡
#查網(wǎng)卡
$ ovs-vsctl show
5a977fc5-4fdf-4fc7-aea3-a7341a305db1
Bridge br-int
fail_mode: secure
Port "tap53eeb988-c7"
tag: 4
Interface "tap53eeb988-c7"
Port "int-br-bo8eb174"
Interface "int-br-bo8eb174"
type: patch
options: {peer="phy-br-bo8eb174"}
Port br-int
Interface br-int
type: internal
Bridge br-bond_vmouter
fail_mode: secure
Port "phy-br-bo8eb174"
Interface "phy-br-bo8eb174"
type: patch
options: {peer="int-br-bo8eb174"}
Port bond_vmouter
Interface bond_vmouter
Port br-bond_vmouter
Interface br-bond_vmouter
type: internal
ovs_version: "2.5.0"
#查端口
$ ovs-ofctl show br-int
FPT_FEATURES_REPLY (xid=0x2): dpid:00005adafb219b49
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
1(int-br-bo8eb174): addr:ae:1f:61:28:e4:96
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
9(tap53eeb988-c7): addr:fe:16:3e:e5:d9:00
config: 0
state: 0
current: 10MB-FD COPPER
speed: 10 Mbps now, 0 Mbps max
LOCAL(br-int): addr:5a:da:fb:21:9b:49
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
對tap53eeb988-c7創(chuàng)建一條Qos柳譬，其中qos0隊(duì)列限速最小700Mbps,最大800Mbps
$ ovs-vsctl -- set port tap53eeb988-c7 qos=@newqos \
-- --id=@newqos create qos type=linux-htb other-config:max-rate=800000000 queues=0=@q0 \
-- --id=@q0 create queue other-config:min-rate=700000000 other-config:max-rate=800000000
查當(dāng)前流表規(guī)則
$ ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
cookie=0xb01c77077412cf51, duration=4216756.364s, table=0, n_packets=390420, n_bytes=36921445, idle_age=0, hard_age=65534, priority=2,in_port=1 actions=drop
cookie=0xb01c77077412cf51, duration=1554953.959s, table=0, n_packets=620904369, n_bytes=519069450871, idle_age=1, hard_age=65534, priority=9,in_port=19 actions=resubmit(,25)
cookie=0xb01c77077412cf51, duration=4216750.815s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=10,icmp6,in_port=9,icmp_type=136 actions=resubmit(,24)
cookie=0xb01c77077412cf51, duration=4216750.804s, table=0, n_packets=82396, n_bytes=3460632, idle_age=2242, hard_age=65534, priority=10,arp,in_port=9 actions=resubmit(,24)
cookie=0xb01c77077412cf51, duration=4216751.878s, table=0, n_packets=1037226147, n_bytes=4734216312377, idle_age=0, hard_age=65534, priority=3,in_port=1,dl_vlan=332 actions=mod_vlan_vid:4,NORMAL
cookie=0xb01c77077412cf51, duration=4216756.481s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=NORMAL
cookie=0xb01c77077412cf51, duration=4216756.473s, table=23, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0xb01c77077412cf51, duration=4216750.821s, table=24, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,icmp6,in_port=9,icmp_type=136,nd_target=fe80::f816:3eff:fee5:d900 actions=NORMAL
cookie=0xb01c77077412cf51, duration=4216750.809s, table=24, n_packets=82154, n_bytes=3450468, idle_age=2242, hard_age=65534, priority=2,arp,in_port=9,arp_spa=10.16.32.40 actions=resubmit(,25)
cookie=0xb01c77077412cf51, duration=4216756.466s, table=24, n_packets=1556, n_bytes=65352, idle_age=5669, hard_age=65534, priority=0 actions=drop
cookie=0xb01c77077412cf51, duration=4215276.624s, table=25, n_packets=1051294240, n_bytes=707172859381, idle_age=20, hard_age=65534, priority=2,in_port=9,dl_src=fa:16:3e:e5:d9:00 actions=NORMAL
從流表里面可以看到通過in_port=9的報(bào)文在table 25里面處理，那么問題就很簡單了续镇，修改table25美澳，將qos隊(duì)列規(guī)則應(yīng)用到in_port=9上就可以了，操作如下：
$ ovs-ofctl mod-flows br-int "table=25, n_packets=1051294240, n_bytes=707172859381, idle_age=20, hard_age=65534, priority=2,in_port=9,dl_src=fa:16:3e:e5:d9:00 actions=set_queue:0,NORMAL"
這個時候再觀察下虛擬機(jī)的監(jiān)控
很好摸航，已經(jīng)成功限制住了制跟。
那么如何查詢qos相關(guān)信息呢？
查看網(wǎng)卡屬性
$ ovs-vsctl list port tap53eeb988-c7
_uuid : 4712ae65-bced-4ee3-bf7d-3b7fa1e52bb7
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
external_ids : {}
fake_bridge : false
interfaces : [1fe8bb0a-6383-45ba-bc86-46e1de03f4e0]
lacp : []
mac : []
name : "tap53eeb988-c7"
other_config : {net_uuid="ea7d53f9-45c6-4027-98b5-23053d10373b", network_type=vlan, physical_network="physnet1", segmentation_id="332", tag="4"}
qos : 82bd0134-4e76-405a-ac1d-22b4ea43e55a
rstp_statistics : {}
rstp_status : {}
statistics : {}
status : {}
tag : 4
trunks : []
vlan_mode : []
這個82bd0134-4e76-405a-ac1d-22b4ea43e55a就是OVS里面QOS的uuid
查看QOS屬性
$ ovs-vsctl list qos 82bd0134-4e76-405a-ac1d-22b4ea43e55a
_uuid : 82bd0134-4e76-405a-ac1d-22b4ea43e55a
external_ids : {}
other_config : {max-rate="800000000"}
queues : {0=cc4e5d2e-2dbb-4e5b-a682-d6a28bd7b743}
type : linux-htb
刪除QOS并清除網(wǎng)卡QOS
$ ovs-vsctl -- destroy QoS 82bd0134-4e76-405a-ac1d-22b4ea43e55a -- clear Port tap53eeb988 qos
libvirtd限速接口
Libvirtd默認(rèn)提供domiftune限制網(wǎng)卡流量
查看虛機(jī)接口的限速設(shè)置
$ virsh domiftune 4ffbd71f-3324-4500-8636-f9a275b6e479 tap53eeb988
設(shè)置虛機(jī)接口限速
$ virsh domiftune 4ffbd71f-3324-4500-8636-f9a275b6e479 tap53eeb988 --inbound 700000,800000,800000 --outbount 700000,800000,800000 --live
單位如下
average bandwidth kilobytes/second
peak bandwidth kilobytes/second
burst size kilobytes
實(shí)際限速值average, 峰值peak和突發(fā)值burst是可以合理計(jì)算出來的
建議的值:
peak=1.5*average
burst=peak/8*2=3average/8
這里要注意的是domiftune只針對網(wǎng)絡(luò)模式為nat忙厌，route等方式凫岖，對模型為bridge, passthrough, private,和hostdev是不支持限制的。
The <bandwidth> element allows setting quality of service for a particular network (since 0.9.4). Setting bandwidth for a network is supported only for networks with a <forward> mode of route, nat, or no mode at all (i.e. an "isolated" network). Setting bandwidth is not supported for forward modes of bridge, passthrough, private, or hostdev. Attempts to do this will lead to a failure to define the network or to create a transient network.
Neutron QosPolicy
這里很慚愧逢净，線上OpenStack的虛擬機(jī)Neutron居然沒開啟QOS驅(qū)動哥放，也怪自己當(dāng)時急于上線，沒有考慮周到〉粒現(xiàn)在再修改的怕影響線上環(huán)境甥雕，故而不敢在上面嘗試，日后測試環(huán)境通過了胀茵，再更新文檔不遲社露。
因?yàn)镹eutron已經(jīng)封裝好了qos的實(shí)現(xiàn)，那么我們拿過來直接用就好了琼娘。
更改Neutron配置文件
neutron.conf
service_plugins = neutron.services.qos.qos_plugin.QoSPlugin
/etc/neutron/plugins/ml2/ml2_conf.ini
[ml2]
extension_drivers=qos
[agent]
extensions=qos
創(chuàng)建一個policy
$ neutron qos-policy-create test1
+-----------------+--------------------------------------+
| Field | Value |
+-----------------+--------------------------------------+
| created_at | 2017-02-27T15:56:11Z |
| description | |
| id |82bd0114-4e76-40da-ac1d-22bbea43e55a |
| name | test1 |
| revision_number | 1 |
| rules | |
| shared | False |
| tenant_id | b61372588a7e4475bc8ecdbaee3fa340 |
| updated_at | 2017-02-27T15:56:11Z |
+-----------------+--------------------------------------+
添加限速rule
$ neutron qos-bandwidth-limit-rule-createbw-limiter --max-kbps 700000 --max-burst-kbps 560000
Created a new bandwidth_limit_rule:
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| id |4d65fbdf-6c08-456b-8545-6f6339f34881 |
| max_burst_kbps | 560000 |
| max_kbps | 700000 |
+----------------+--------------------------------------+
綁定Port
$ neutron port-update <port_id> --qos-policy test
綁定Network
$ neutron net-update <network_id> --qos-policy test
