X710-DA4的l3fwd測(cè)試

X710-DA4有四個(gè)10GE口

兩臺(tái)DELL 730(支持PCIE3.0)各插一張X710-DA4網(wǎng)卡

一臺(tái)用于發(fā)包真屯，一臺(tái)用于l3fwd

ubuntu-1(pktgen-dpdk,ubuntu17.10)

68:05:CA:32:02:F0

68:05:CA:32:02:F1

68:05:CA:32:02:F2

68:05:CA:32:02:F3

ubuntu-2(l3fwd? dpdk17.11,ubuntu17.10)

68:05:CA:32:03:18

68:05:CA:32:03:19

68:05:CA:32:03:1A

68:05:CA:32:03:1B

端口對(duì)連

68:05:CA:32:02:F0----68:05:CA:32:03:18

68:05:CA:32:02:F1----68:05:CA:32:03:19

68:05:CA:32:02:F2----68:05:CA:32:03:1A

68:05:CA:32:02:F3----68:05:CA:32:03:1B

驅(qū)動(dòng)與固件版本

root@ubuntu-2:~# ethtool? -i enp5s0f0

driver: i40e

version: 2.4.6

firmware-version: 6.01 0x80003494 1.1747.0

expansion-rom-version:

bus-info: 0000:05:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

查看網(wǎng)卡所在的cpu socket號(hào)((或者用 lstopo-no-graphics鳞滨，見(jiàn)文章最后))

root@ubuntu-2:~# cat /sys/class/net/enp5s0f0/device/numa_node

0

root@ubuntu-2:~# cat /sys/class/net/enp5s0f1/device/numa_node?

0

root@ubuntu-2:~# cat /sys/class/net/enp5s0f2/device/numa_node?

0

root@ubuntu-2:~# cat /sys/class/net/enp5s0f3/device/numa_node?

0

查看cpu core和lcpu與cpu socket的對(duì)應(yīng)關(guān)系

root@ubuntu-2:~# cat start-l3fwd.sh

export DPDK_DIR=/root/dpdk/dpdk-17.11

export DPDK_TARGET=x86_64-native-linuxapp-gcc

export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET

mkdir -p /dev/hugepages

mount -t hugetlbfs hugetlbfs /dev/hugepages

modprobe uio

insmod $DPDK_BUILD/kmod/igb_uio.ko

export RTE_SDK=$DPDK_DIR

export RTE_TARGET=$DPDK_TARGET

cd /root/dpdk/

####X710-DA4(將lcpu分配在網(wǎng)卡所在的socket為高性能且警，否則為低性能,以下用兩個(gè)l3fwd分別進(jìn)行了對(duì)比測(cè)試,先用lcpu=3,5,6,9,15,17,19,21測(cè)試低性能把敢，再用lcpu=2,4,6,8,14,16,18,20測(cè)試高性能)

$DPDK_DIR/usertools/dpdk-devbind.py -b igb_uio 0000:05:00.0 0000:05:00.1 0000:05:00.2 0000:05:00.3

./l3fwd -l 3,5,7,9,15,17,19,21 -n 4 --proc-type auto --socket-mem 2048,2048? --huge-dir /dev/hugepages -- -p 0xf? -L --config="(0,0,3),(0,1,15),(1,0,5),(1,1,17),(2,0,7),(2,1,19),(3,0,9),(3,1,21)"

#./l3fwd -l 2,4,6,8,14,16,18,20 -n 4 --proc-type auto --socket-mem 2048,2048? --huge-dir /dev/hugepages -- -p 0xf? -L --config="(0,0,2),(0,1,14),(1,0,4),(1,1,16),(2,0,6),(2,1,18),(3,0,8),(3,1,20)"

執(zhí)行l(wèi)3fwd

注意路由及目標(biāo)MAC

root@ubuntu-2:~# sh start-l3fwd.sh? ?

EAL: Detected 24 lcore(s)

EAL: Auto-detected process type: PRIMARY

EAL: No free hugepages reported in hugepages-1048576kB

EAL: Probing VFIO support...

EAL: PCI device 0000:05:00.0 on NUMA socket 0

EAL:? probe driver: 8086:1572 net_i40e

PMD: Global register is changed during enable FDIR flexible payload

PMD: Global register is changed during support QinQ parser

PMD: Global register is changed during configure hash input set

PMD: Global register is changed during configure fdir mask

PMD: Global register is changed during configure hash mask

PMD: Global register is changed during support QinQ cloud filter

PMD: Global register is changed during support TPID configuration

EAL: PCI device 0000:05:00.1 on NUMA socket 0

EAL:? probe driver: 8086:1572 net_i40e

PMD: Global register is changed during enable FDIR flexible payload

PMD: Global register is changed during support QinQ parser

PMD: Global register is changed during configure hash input set

PMD: Global register is changed during configure fdir mask

PMD: Global register is changed during configure hash mask

PMD: Global register is changed during support QinQ cloud filter

PMD: Global register is changed during support TPID configuration

EAL: PCI device 0000:05:00.2 on NUMA socket 0

EAL:? probe driver: 8086:1572 net_i40e

PMD: Global register is changed during enable FDIR flexible payload

PMD: Global register is changed during support QinQ parser

PMD: Global register is changed during configure hash input set

PMD: Global register is changed during configure fdir mask

PMD: Global register is changed during configure hash mask

PMD: Global register is changed during support QinQ cloud filter

PMD: Global register is changed during support TPID configuration

EAL: PCI device 0000:05:00.3 on NUMA socket 0

EAL:? probe driver: 8086:1572 net_i40e

PMD: Global register is changed during enable FDIR flexible payload

PMD: Global register is changed during support QinQ parser

PMD: Global register is changed during configure hash input set

PMD: Global register is changed during configure fdir mask

PMD: Global register is changed during configure hash mask

PMD: Global register is changed during support QinQ cloud filter

PMD: Global register is changed during support TPID configuration

EAL: PCI device 0000:07:00.0 on NUMA socket 0

EAL:? probe driver: 8086:1572 net_i40e

EAL: PCI device 0000:07:00.1 on NUMA socket 0

EAL:? probe driver: 8086:1572 net_i40e

EAL: PCI device 0000:84:00.0 on NUMA socket 1

EAL:? probe driver: 8086:10fb net_ixgbe

EAL: PCI device 0000:84:00.1 on NUMA socket 1

EAL:? probe driver: 8086:10fb net_ixgbe

L3FWD: Longest-prefix match selected

Initializing port 0 ... Creating queues: nb_rxq=2 nb_txq=8...Address:68:05:CA:32:03:18, Destination:02:00:00:00:00:00, Allocated mbuf pool on socket 1

LPM: Adding route 0x01010100 / 24 (0)

LPM: Adding route 0x02010100 / 24 (1)

LPM: Adding route 0x03010100 / 24 (2)

LPM: Adding route 0x04010100 / 24 (3)

LPM: Adding route IPV6 / 48 (0)

LPM: Adding route IPV6 / 48 (1)

LPM: Adding route IPV6 / 48 (2)

LPM: Adding route IPV6 / 48 (3)

txq=3,0,1 txq=5,1,1 txq=7,2,1 txq=9,3,1 txq=15,4,1 txq=17,5,1 txq=19,6,1 txq=21,7,1

Initializing port 1 ... Creating queues: nb_rxq=2 nb_txq=8...Address:68:05:CA:32:03:19, Destination:02:00:00:00:00:01, txq=3,0,1 txq=5,1,1 txq=7,2,1 txq=9,3,1 txq=15,4,1 txq=17,5,1 txq=19,6,1 txq=21,7,1

Initializing port 2 ... Creating queues: nb_rxq=2 nb_txq=8... Address:68:05:CA:32:03:1A, Destination:02:00:00:00:00:02, txq=3,0,1 txq=5,1,1 txq=7,2,1 txq=9,3,1 txq=15,4,1 txq=17,5,1 txq=19,6,1 txq=21,7,1

Initializing port 3 ... Creating queues: nb_rxq=2 nb_txq=8...Address:68:05:CA:32:03:1B, Destination:02:00:00:00:00:03, txq=3,0,1 txq=5,1,1 txq=7,2,1 txq=9,3,1 txq=15,4,1 txq=17,5,1 txq=19,6,1 txq=21,7,1

Initializing rx queues on lcore 3 ... rxq=0,0,1

Initializing rx queues on lcore 5 ... rxq=1,0,1

Initializing rx queues on lcore 7 ... rxq=2,0,1

Initializing rx queues on lcore 9 ... rxq=3,0,1

Initializing rx queues on lcore 15 ... rxq=0,1,1

Initializing rx queues on lcore 17 ... rxq=1,1,1

Initializing rx queues on lcore 19 ... rxq=2,1,1

Initializing rx queues on lcore 21 ... rxq=3,1,1

Checking link statusdone

Port0 Link Up. Speed 10000 Mbps -full-duplex

Port1 Link Up. Speed 10000 Mbps -full-duplex

Port2 Link Up. Speed 10000 Mbps -full-duplex

Port3 Link Up. Speed 10000 Mbps -full-duplex

L3FWD: entering main loop on lcore 5

L3FWD:? -- lcoreid=5 portid=1 rxqueueid=0

L3FWD: entering main loop on lcore 7

L3FWD:? -- lcoreid=7 portid=2 rxqueueid=0

L3FWD: entering main loop on lcore 15

L3FWD: entering main loop on lcore 21

L3FWD:? -- lcoreid=21 portid=3 rxqueueid=1

L3FWD: entering main loop on lcore 9

L3FWD:? -- lcoreid=9 portid=3 rxqueueid=0

L3FWD: entering main loop on lcore 19

L3FWD:? -- lcoreid=19 portid=2 rxqueueid=1

L3FWD:? -- lcoreid=15 portid=0 rxqueueid=1

L3FWD: entering main loop on lcore 17

L3FWD:? -- lcoreid=17 portid=1 rxqueueid=1

L3FWD: entering main loop on lcore 3

L3FWD:? -- lcoreid=3 portid=0 rxqueueid=0

說(shuō)明:

LPM: Adding route 0x01010100 / 24 (0)?

LPM: Adding route 0x02010100 / 24 (1)

LPM: Adding route 0x03010100 / 24 (2)

LPM: Adding route 0x04010100 / 24 (3)

使用的LPM最長(zhǎng)匹配模式

數(shù)據(jù)包經(jīng)端口0發(fā)送出去,可以到達(dá)1.1.1.0/24網(wǎng)段,目標(biāo)mac為02:00:00:00:00:00

數(shù)據(jù)包經(jīng)端口1發(fā)送出去,可以到達(dá)2.1.1.0/24網(wǎng)段,目標(biāo)mac為02:00:00:00:00:01

數(shù)據(jù)包經(jīng)端口2發(fā)送出去,可以到達(dá)3.1.1.0/24網(wǎng)段,目標(biāo)mac為02:00:00:00:00:02

數(shù)據(jù)包經(jīng)端口3發(fā)送出去,可以到達(dá)4.1.1.0/24網(wǎng)段,目標(biāo)mac為02:00:00:00:00:03

發(fā)包方

root@ubuntu-1:~# cat start-pktgen-dpdk-x710.sh

export DPDK_DIR=/root/dpdk/dpdk-17.11

export DPDK_TARGET=x86_64-native-linuxapp-gcc

export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET

mkdir -p /dev/hugepages

mount -t hugetlbfs hugetlbfs /dev/hugepages

modprobe uio

insmod $DPDK_BUILD/kmod/igb_uio.ko

export RTE_SDK=$DPDK_DIR

export RTE_TARGET=$DPDK_TARGET

cd /root/pktgen-3.4.9/

$DPDK_DIR/usertools/dpdk-devbind.py -b igb_uio 0000:05:00.0 0000:05:00.1 0000:05:00.2 0000:05:00.3

./pktgen? -l 0,2-9? -n 4 --proc-type auto --socket-mem 2048,2048? --huge-dir /dev/hugepages? -- -P -T -m '[2:3].0,[4:5].1,[6:7].2,[8:9].3'?

準(zhǔn)備發(fā)包

root@ubuntu-1:~# sh start-pktgen-dpdk-x710.sh

發(fā)包配置

set 0 src ip 1.1.1.241/24

set 0 src mac 02:00:00:00:00:00

set 0 dst ip? 2.1.1.242

set 0 dst mac 68:05:CA:32:03:18

set 1 src ip? 2.1.1.242/24

set 1 src mac 02:00:00:00:00:01

set 1 dst ip? 1.1.1.241

set 1 dst mac 68:05:CA:32:03:19

set 2 src ip? 3.1.1.241/24

set 2 src mac 02:00:00:00:00:02

set 2 dst ip? 4.1.1.242

set 2 dst mac 68:05:CA:32:03:1A

set 3 src ip? 4.1.1.242/24

set 3 src mac 02:00:00:00:00:03

set 3 dst ip? 3.1.1.241

set 3 dst mac 68:05:CA:32:03:1B

#啟動(dòng)端口發(fā)包(全雙向)

start 0

start 1

start 2

start 3

路由表的設(shè)計(jì)參見(jiàn)intel

https://software.intel.com/en-us/articles/introduction-to-the-dpdk-sample-applications

低性能(lcpu=3,5,7,9,15,17,19,21)----10.55Mpps

高性能(lcpu=2,4,6,8,14,16,18,20)----37.3Mpps

結(jié)論:

l3fwd轉(zhuǎn)發(fā)激捏，lcpu的綁定一定要在網(wǎng)卡所在的socket上,否則會(huì)導(dǎo)致低性能

用lstopo可以查看那個(gè)pci設(shè)備和那個(gè)scoket直連

apt-get install hwloc

root@ubuntu-1:~# lstopo-no-graphics

Machine (63GB total)

? NUMANode L#0 (P#0 31GB)

? ? Package L#0 + L3 L#0 (15MB)

? ? ? L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0

? ? ? ? PU L#0 (P#0)

? ? ? ? PU L#1 (P#12)

? ? ? L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1

? ? ? ? PU L#2 (P#2)

? ? ? ? PU L#3 (P#14)

? ? ? L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2

? ? ? ? PU L#4 (P#4)

? ? ? ? PU L#5 (P#16)

? ? ? L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3

? ? ? ? PU L#6 (P#6)

? ? ? ? PU L#7 (P#18)

? ? ? L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4

? ? ? ? PU L#8 (P#8)

? ? ? ? PU L#9 (P#20)

? ? ? L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5

? ? ? ? PU L#10 (P#10)

? ? ? ? PU L#11 (P#22)

? ? HostBridge L#0

? ? ? PCIBridge

? ? ? ? PCI 1000:005d

? ? ? ? ? Block(Disk) L#0 "sda"

? ? ? ? ? Block(Disk) L#1 "sdb"

? ? ? PCIBridge

? ? ? ? PCI 8086:1572

? ? ? ? ? Net L#2 "enp5s0f0"

? ? ? ? PCI 8086:1572

? ? ? ? ? Net L#3 "enp5s0f1"

? ? ? ? PCI 8086:1572

? ? ? ? ? Net L#4 "enp5s0f2"

? ? ? ? PCI 8086:1572

? ? ? ? ? Net L#5 "enp5s0f3"

? ? ? PCIBridge

? ? ? ? PCI 14e4:168a

? ? ? ? ? Net L#6 "eno1"

? ? ? ? PCI 14e4:168a

? ? ? ? ? Net L#7 "eno2"

? ? ? ? PCI 14e4:168a

? ? ? ? ? Net L#8 "eno3"

? ? ? ? PCI 14e4:168a

? ? ? ? ? Net L#9 "eno4"

? ? ? PCIBridge

? ? ? ? PCI 8086:1572

? ? ? ? ? Net L#10 "enp7s0f0"

? ? ? ? PCI 8086:1572

? ? ? ? ? Net L#11 "enp7s0f1"

? ? ? PCI 8086:8d62

? ? ? PCIBridge

? ? ? ? PCIBridge

? ? ? ? ? PCIBridge

? ? ? ? ? ? PCIBridge

? ? ? ? ? ? ? PCI 102b:0534

? ? ? ? ? ? ? ? GPU L#12 "card0"

? ? ? ? ? ? ? ? GPU L#13 "controlD64"

? ? ? PCI 8086:8d02

? NUMANode L#1 (P#1 31GB)

? ? Package L#1 + L3 L#1 (15MB)

? ? ? L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6

? ? ? ? PU L#12 (P#1)

? ? ? ? PU L#13 (P#13)

? ? ? L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7

? ? ? ? PU L#14 (P#3)

? ? ? ? PU L#15 (P#15)

? ? ? L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8

? ? ? ? PU L#16 (P#5)

? ? ? ? PU L#17 (P#17)

? ? ? L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9

? ? ? ? PU L#18 (P#7)

? ? ? ? PU L#19 (P#19)

? ? ? L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10

? ? ? ? PU L#20 (P#9)

? ? ? ? PU L#21 (P#21)

? ? ? L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11

? ? ? ? PU L#22 (P#11)

? ? ? ? PU L#23 (P#23)

? ? HostBridge L#9

? ? ? PCIBridge

? ? ? ? PCI 8086:10fb

? ? ? ? ? Net L#14 "enp132s0f0"

? ? ? ? PCI 8086:10fb

? ? ? ? ? Net L#15 "enp132s0f1"

? ? ? ? 4 x { PCI 8086:10ed }

最后編輯于：2018.03.24 11:59:54

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者

人面猴
序言：七十年代末鸿竖，一起剝皮案震驚了整個(gè)濱河市武花，隨后出現(xiàn)的幾起案子圆凰，更是在濱河造成了極大的恐慌，老刑警劉巖体箕，帶你破解...
沈念sama閱讀 219,366評(píng)論 6贊 508
死咒
序言：濱河連續(xù)發(fā)生了三起死亡事件专钉，死亡現(xiàn)場(chǎng)離奇詭異挑童，居然都是意外死亡，警方通過(guò)查閱死者的電腦和手機(jī)跃须，發(fā)現(xiàn)死者居然都...
沈念sama閱讀 93,521評(píng)論 3贊 395
救了他兩次的神仙讓他今天三更去死
文/潘曉璐我一進(jìn)店門站叼，熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)，“玉大人菇民，你說(shuō)我怎么就攤上這事尽楔。” “怎么了第练？”我有些...
開封第一講書人閱讀 165,689評(píng)論 0贊 356
道士緝兇錄：失蹤的賣姜人
文/不壞的土叔我叫張陵阔馋，是天一觀的道長(zhǎng)。經(jīng)常有香客問(wèn)我娇掏，道長(zhǎng)呕寝，這世上最難降的妖魔是什么？我笑而不...
開封第一講書人閱讀 58,925評(píng)論 1贊 295
?港島之戀（遺憾婚禮）
正文為了忘掉前任婴梧，我火速辦了婚禮下梢，結(jié)果婚禮上，老公的妹妹穿的比我還像新娘塞蹭。我一直安慰自己怔球，他們只是感情好，可當(dāng)我...
茶點(diǎn)故事閱讀 67,942評(píng)論 6贊 392
惡毒庶女頂嫁案：這布局不是一般人想出來(lái)的
文/花漫我一把揭開白布浮还。她就那樣靜靜地躺著，像睡著了一般闽巩。火紅的嫁衣襯著肌膚如雪钧舌。梳的紋絲不亂的頭發(fā)上，一...
開封第一講書人閱讀 51,727評(píng)論 1贊 305
城市分裂傳說(shuō)
那天涎跨，我揣著相機(jī)與錄音洼冻，去河邊找鬼。笑死隅很，一個(gè)胖子當(dāng)著我的面吹牛撞牢，可吹牛的內(nèi)容都是我干的。我是一名探鬼主播叔营，決...
沈念sama閱讀 40,447評(píng)論 3贊 420
雙鴛鴦連環(huán)套：你想象不到人心有多黑
文/蒼蘭香墨我猛地睜開眼屋彪，長(zhǎng)吁一口氣：“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼！你這毒婦竟也來(lái)了绒尊？” 一聲冷哼從身側(cè)響起畜挥，我...
開封第一講書人閱讀 39,349評(píng)論 0贊 276
萬(wàn)榮殺人案實(shí)錄
序言：老撾萬(wàn)榮一對(duì)情侶失蹤，失蹤者是張志新（化名）和其女友劉穎婴谱，沒(méi)想到半個(gè)月后蟹但，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體躯泰，經(jīng)...
沈念sama閱讀 45,820評(píng)論 1贊 317
?護(hù)林員之死
正文獨(dú)居荒郊野嶺守林人離奇死亡，尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛以下內(nèi)容為張勛視角年9月15日...
茶點(diǎn)故事閱讀 37,990評(píng)論 3贊 337
?白月光啟示錄
正文我和宋清朗相戀三年华糖，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了麦向。大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
茶點(diǎn)故事閱讀 40,127評(píng)論 1贊 351
活死人
序言：一個(gè)原本活蹦亂跳的男人離奇死亡客叉，死狀恐怖诵竭，靈堂內(nèi)的尸體忽然破棺而出，到底是詐尸還是另有隱情十办，我是刑警寧澤秀撇，帶...
沈念sama閱讀 35,812評(píng)論 5贊 346
?日本核電站爆炸內(nèi)幕
正文年R本政府宣布，位于F島的核電站向族，受9級(jí)特大地震影響呵燕，放射性物質(zhì)發(fā)生泄漏。R本人自食惡果不足惜件相，卻給世界環(huán)境...
茶點(diǎn)故事閱讀 41,471評(píng)論 3贊 331
男人毒藥：我在死后第九天來(lái)索命
文/蒙蒙一再扭、第九天我趴在偏房一處隱蔽的房頂上張望。院中可真熱鬧夜矗，春花似錦泛范、人聲如沸。這莊子的主人今日做“春日...
開封第一講書人閱讀 32,017評(píng)論 0贊 22
一樁弒父案罢荡，背后竟有這般陰謀
文/蒼蘭香墨我抬頭看了看天上的太陽(yáng)。三九已至对扶，卻和暖如春区赵，著一層夾襖步出監(jiān)牢的瞬間，已是汗流浹背浪南。一陣腳步聲響...
開封第一講書人閱讀 33,142評(píng)論 1贊 272
情欲美人皮
我被黑心中介騙來(lái)泰國(guó)打工笼才，沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留，地道東北人络凿。一個(gè)月前我還...
沈念sama閱讀 48,388評(píng)論 3贊 373
代替公主和親
正文我出身青樓骡送，卻偏偏與公主長(zhǎng)得像，于是被迫代替她去往敵國(guó)和親絮记。傳聞我的和親對(duì)象是個(gè)殘疾皇子摔踱，可洞房花燭夜當(dāng)晚...
茶點(diǎn)故事閱讀 45,066評(píng)論 2贊 355