By Toradex秦海
1).?簡介
DPDK (Data Plane Development Kit)?軟件是一組用戶空間庫和驅(qū)動程序屏箍，可加速在所有主要?CPU?架構上運行的網(wǎng)絡數(shù)據(jù)包處理工作負載耀鸦，以便提升整個網(wǎng)絡數(shù)據(jù)服務的QoS捶枢。其最早由?Intel?大約?2010年創(chuàng)建，后由6WIND公司發(fā)展為開源社區(qū)版本后火邓，極大的拓展了DPDK的影響力丹弱，現(xiàn)在DPDI已經(jīng)是是?Linux?基金會旗下的開源項目之一，其支持涵蓋了主流的CPU架構和繁多的NIC驅(qū)動以及虛擬化平臺支持铲咨。
DPDK使用了輪詢?(polling)?而不是中斷來處理數(shù)據(jù)包躲胳。在收到數(shù)據(jù)包時，經(jīng)?DPDK?重載的網(wǎng)卡驅(qū)動不會通過中斷通知?CPU?纤勒，而是直接將數(shù)據(jù)包存入內(nèi)存坯苹，交付應用層軟件通過?DPDK?提供的接口來直接處理，這樣節(jié)省了大量的?CPU?中斷時間和內(nèi)存拷貝時間摇天。
本文所要測試的?NXP iMX8M Mini ARM SoC平臺集成的?FEC NIC?在?DPDK 21.11 LTS發(fā)布版本提供了相關支持粹湃， 具體的?DPDK ENETFEC Poll Mode?驅(qū)動描述以及功能支持情況請見如下?dpdk.org?文檔恐仑，本文就對其進行簡單部署測試。
http://doc.dpdk.org/guides/nics/enetfec.html
本文所演示的平臺來自于ToradexVerdin iMX8M Mini嵌入式平臺再芋，基于?NXP iMX8M Mini?系列?ARM?處理器菊霜，核心為?Cortex-A53坚冀，包含一個支持?DPDK PMD?驅(qū)動的以太網(wǎng)接口济赎。
2).?硬件準備
a).?DUT-A :?Verdin iMX8MM ARM?核心版配合?Dahlia Carrier Board?載板，并連接調(diào)試串口记某，后續(xù)測試中作為DPDK Packet Generator?設備司训。
b).?DUT-B :?Verdin iMX8MM ARM?核心版配合?Verdin Development Board?載板，并連接調(diào)試串口液南，后續(xù)測試中作為DPDK Packet Generator?設備壳猜。
c).?Dahlia?載板?X11千兆網(wǎng)口和?Verdin Development Board X25?千兆網(wǎng)口通過一根網(wǎng)線直通連接。
?
?
3).?軟件準備
a). Toradex Verdin iMXMM?模塊標準?Ycoto Linux BSP中沒有包含?DPDK?相關庫和驅(qū)動滑凉，需要通過?Ycoto Project/Openembedded?編譯環(huán)境來進行添加編譯统扳，生成新的測試?Linux BSP Image。
b).?參考?Ycoto?編譯環(huán)境來下載配置相關編譯環(huán)境畅姊，使用最新的針對?Ycoto Linux BSP 6/Linux Kernel 5.15.x?版本的?branch
---------------------------------------
$ mkdir ${HOME}/oe-core
$ cd ${HOME}/oe-core
$ repo init -u git://git.toradex.com/toradex-manifest.git -b kirkstone-6.x.y -m tdxref/default.xml
$ repo sync
---------------------------------------
c).?由于默認的?oe-core/layers/meta-freescale/recipes-extended/dpdk/?提供的?DPDK 19.11版本還未支持?iMX8MM咒钟，需要參考?NXP i.MX_Yocto_Project_User's_Guide?文檔，下載?NXP imx-6.1.1-1.0.0 branch?代碼來獲取?DPDK 21.11?版本的?BB?描述文件若未。
---------------------------------------
$ mkdir ${HOME}/imx-yocto-bsp
$ cd ${HOME}/imx-yocto-bsp
$ repo init -u https://github.com/nxp-imx/imx-manifest -b imx-linux-langdale -m imx-6.1.1-1.0.0.xml
$ repo sync
---------------------------------------
d).?將?NXP imx-6.1.1-1.0.0 branch?包含的?DPDK 21.11 BB?文件復制到?Toradex Ycoto?環(huán)境
---------------------------------------
$ cd ${HOME}/oe-core
$ mkdir -p layers/meta-toradex-nxp/recipes-extended/dpdk
$ cp ${HOME}/imx-ycoto-bsp_6.1.1-1.0.0/sources/meta-imx/meta-sdk/recipes-extended/dpdk/dpdk* layers/meta-toradex-nxp/recipes-extended/dpdk/
$ $ ls layers/meta-toradex-nxp/recipes-extended/dpdk/
dpdk_21.11.bb ?dpdk.inc
---------------------------------------
e).?修改配置文件后編譯測試?Ycoto Linux BSP Image?文件
---------------------------------------
### Add below to ${HOME}/oe-core/build/conf/local.conf ###
# add dpdk package
IMAGE_INSTALL:append = " dpdk dpdk-examples dpdk-tools"
# accept freescale license
ACCEPT_FSL_EULA = "1"
### Compile BSP Image ###
$ MACHINE=verdin-imx8mm bitbake tdx-reference-multimedia-image
---------------------------------------
f).?參考這里的說明將編譯生成的?Ycoto Linux Image?通過?Toradex Easy Installer?更新到兩個測試平臺的?Verdin iMX8MM?模塊
g).?由于默認?Linux BSP?網(wǎng)口驅(qū)動還是?NXP?標準?Linux FEC Kernel驅(qū)動朱嘴，需要通過修改?Linux Kernel Device Tree?來適配為?DPDK PMD?驅(qū)動。
./?參考這里的說明下載?Linux Kernel?源代碼粗合， 對應?Branch?為?toradex_5.15-2.1.x-imx
./?參考如下?Patch?文件修改?Device Tree?源代碼萍嬉，并重新編譯生成所需的?Device Tree Binary?文件?imx8mm-verdin-wifi-dev.dtb
---------------------------------------
diff --git a/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts b/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts
index ef952021832e..b2d60695ecf9 100644
--- a/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts
+++ b/arch/arm64/boot/dts/freescale/imx8mm-verdin-wifi-dev.dts
@@ -16,3 +16,7 @@
?????????????????????"toradex,verdin-imx8mm",
?????????????????????"fsl,imx8mm";
?};
+
+&fec1 {
+ ???????compatible = "fsl,imx8mm-fec-uio";
+};
---------------------------------------
./?將新的?imx8mm-verdin-wifi-dev.dtb?替換兩個平臺的?Verdin iMX8MM Linux /boot?下默認使用的?Device Tree?文件并重新啟動，查看原本的?eth0?接口已經(jīng)沒有?probe?了
---------------------------------------
root@verdin-imx8mm-07276322:~# dmesg |grep eth
[ ???0.000000] psci: probing for conduit method from DT.
[ ???1.716513]?fec-uio 30be0000.ethernet: UIO device full name imx-fec-uio initialized
[ ???8.382278] using random self ethernet address
[ ???8.387105] using random host ethernet address
root@verdin-imx8mm-07276322:~# ifconfig -a
can0: flags=128<NOARP> ?mtu 16
????????unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 ?txqueuelen 10 ?(UNSPEC)
????????RX packets 0 ?bytes 0 (0.0 B)
????????RX errors 0 ?dropped 0 ?overruns 0 ?frame 0
????????TX packets 0 ?bytes 0 (0.0 B)
????????TX errors 0 ?dropped 0 overruns 0 ?carrier 0 ?collisions 0
????????device interrupt 78 ?
lo: flags=73<UP,LOOPBACK,RUNNING> ?mtu 65536
????????inet 127.0.0.1 ?netmask 255.0.0.0
????????inet6 ::1 ?prefixlen 128 ?scopeid 0x10<host>
????????loop ?txqueuelen 1000 ?(Local Loopback)
????????RX packets 188 ?bytes 13742 (13.4 KiB)
????????RX errors 0 ?dropped 0 ?overruns 0 ?frame 0
????????TX packets 188 ?bytes 13742 (13.4 KiB)
????????TX errors 0 ?dropped 0 overruns 0 ?carrier 0 ?collisions 0
...
---------------------------------------
4).?DPDK?示例應用測試
a).?首先配置?Huge Pages隙疚，可以通過設置下面?U-boot?環(huán)境變量來配置壤追，這樣就無需每次啟動都配置一次
---------------------------------------
### configure huge paces ###
$ fw_setenv?tdxargs?‘default_hugepagesz=2m hugepagesz=2m hugepages=448’
$ reboot
### check huge pages status ###
root@verdin-imx8mm-07276322:~# cat /proc/meminfo |grep -i '^hugepages' ?????????????????????????
HugePages_Total: ????448
HugePages_Free: ?????448
HugePages_Rsvd: ???????0
HugePages_Surp: ???????0
Hugepagesize: ??????2048 kB
---------------------------------------
b).?示例測試1?：?dpdk-testpmd?示例應用測試
./ dpdk-testpmd?應用配合的參數(shù)說明可以參考這里說明
./ DUT-B?通過?dpdk-testpmd?示例應用?Rx?模式作為?Packet Receiver
---------------------------------------
### DUT-B configure to Rx Mode ###
root@verdin-imx8mm-07275765:~#?dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=rxonly???????????????????????????????????????????????????????????????????
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: lsx_pciep:SoC(0xcc000000) not supported
EAL: Scan for (lsx_pciep) bus failed.
EAL: Selected IOVA mode 'PA'
EAL: No available 32768 kB hugepages reported
EAL: No available 64 kB hugepages reported
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Set rxonly packet forwarding mode
Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config p
arameters along with --numa.
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
Configuring Port 0 (socket 0)
Port 0: 01:01:01:01:01:01
Checking link statuses...
Done
### start forward mode ###
testpmd> start
rxonly packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation m
ode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
rxonly packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=1
port 0: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 ?wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 ?wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
---------------------------------------
./ DUT-A?通過?dpdk-testpmd?示例應用?Tx?模式作為?Packet Generator
---------------------------------------
### DUT-A configure to Tx Mode ###
root@verdin-imx8mm-07276322:~#?dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=txonly
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: lsx_pciep:SoC(0xcc000000) not supported
EAL: Scan for (lsx_pciep) bus failed.
EAL: Selected IOVA mode 'PA'
EAL: No available 32768 kB hugepages reported
EAL: No available 64 kB hugepages reported
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Set txonly packet forwarding mode
Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config p
arameters along with --numa.
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
Configuring Port 0 (socket 0)
Port 0: 01:01:01:01:01:01
Checking link statuses...
Done
### start forward mode ###
testpmd> start
txonly packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation m
ode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
txonly packet forwarding packets/burst=32
packet len=64 - nb packet segments=1
nb forwarding cores=1 - nb forwarding ports=1
port 0: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 ?wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 ?wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
---------------------------------------
./?測試一段時間后按順序分別停止?DUT-A?和?DUT-B，查看統(tǒng)計結果供屉，顯示收發(fā)一致大诸，工作正常
---------------------------------------
### Tx statistic on DUT-A ###
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
---------------------- Forward statistics for port 0 ?----------------------
RX-packets: 0 ?????????????RX-dropped: 0 ????????????RX-total: 0
TX-packets: 23614688 ??????TX-dropped: 0 ????????????TX-total:?23614688
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 0 ?????????????RX-dropped: 0 ????????????RX-total: 0
TX-packets: 23614688 ??????TX-dropped: 0 ????????????TX-total: 23614688
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
### Rx statistic on DUT-B ###
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
---------------------- Forward statistics for port 0 ?----------------------
RX-packets: 23614688 ??????RX-dropped: 0 ????????????RX-total: 23614688
TX-packets: 0 ?????????????TX-dropped: 0 ????????????TX-total: 0
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 23614688 ??????RX-dropped: 0 ????????????RX-total: 23614688
TX-packets: 0 ?????????????TX-dropped: 0 ????????????TX-total: 0
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------
c).?示例測試2?：?dpdk-testpmd Tx Mode?配合?dpdk-l2fwd?示例應用測試
./ dpdk-l2fwd?應用配合的參數(shù)說明可以參考這里說明
./ DUT-B?通過?dpdk-l2fwd?配置為?L2 forward?模式
---------------------------------------
root@verdin-imx8mm-07275765:~# /usr/share/dpdk/examples/dpdk-l2fwd -c 0x1 -n 1 --vdev 'net_enetfec' -- -p 0x1 -P
---------------------------------------
//?注意使用?-P?參數(shù)來接受全部?MAC Address的包，否則不屬于本機的會被自動丟棄
./ DUT-A?如示例?1?通過?dpdk-testpmd?示例應用?Tx?模式作為?Packet Generator?并?start?開始發(fā)包測試
---------------------------------------
root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1?--txd=1 --burst=1--forward-mode=txonly
...
testpmd> start
...
---------------------------------------
//?注意贯卦，使用?--txd?和?--burst?來控制發(fā)包的速率资柔，太快則?dpdk-l2fwd?無法及時處理
./?測試一段時間后停止?DUT-A?發(fā)包，查看統(tǒng)計結果撵割，顯示收發(fā)一致贿堰，同時?DUT-B L2 IO?轉發(fā)也一致，工作正常
---------------------------------------
### Tx statistic on DUT-A ###
??---------------------- Forward statistics for port 0 ?----------------------
??RX-packets: 0 ?????????????RX-dropped: 0 ????????????RX-total: 0
TX-packets:?13839386???????TX-dropped: 41509766 ?????TX-total: 55349152
??----------------------------------------------------------------------------
??+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
??RX-packets: 0 ?????????????RX-dropped: 0 ????????????RX-total: 0
??TX-packets: 13839386 ??????TX-dropped: 83019532 ?????TX-total: 96858918
??++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
### IO statistic on DUT-B ###
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: ????????????????13839386
Packets received: ????????????13839386
Packets dropped: ????????????????????0
Aggregate statistics ===============================
Total packets sent: ??????????13839386
Total packets received: ??????13839386
Total packets dropped: ??????????????0
====================================================
---------------------------------------
d).?示例測試3?：?dpdk-testpmd IO Mode?示例應用測試
./ DUT-B?通過?dpdk-testpmd?配置為?IO forward?模式并?start?開始準備轉發(fā)從?DUT-A?發(fā)送的包
---------------------------------------
root@verdin-imx8mm-07275765:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=io
...
testpmd> start
...
---------------------------------------
./ DUT-A?通過?dpdk-testpmd?配置為?IO forward?模式并通過?start tx_first開始進行先發(fā)一個包然后再?IO?收包的測試
---------------------------------------
root@verdin-imx8mm-07276322:~# dpdk-testpmd -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=io
...
testpmd> start tx_first
...
---------------------------------------
./?測試一段時間后按順序分別停止?DUT-A?和?DUT-B啡彬，查看統(tǒng)計結果羹与，DUT-A?因為先發(fā)一組?Burst?包（?Burst?數(shù)值默認是32）故硅，因此?Tx?比?Rx?多32；同時?DUT-B IO?轉發(fā)一致且和?DUT-A Tx?一致纵搁，工作正常
---------------------------------------
### IO statistic on DUT-A ###
??---------------------- Forward statistics for port 0 ?----------------------
??RX-packets: 10381092 ??????RX-dropped: 0 ????????????RX-total:?10381092
TX-packets: 10381124 ??????TX-dropped: 0 ????????????TX-total:?10381124
??----------------------------------------------------------------------------
??+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
??RX-packets: 10381092 ??????RX-dropped: 0 ????????????RX-total: 10381092
??TX-packets: 10381124 ??????TX-dropped: 0 ????????????TX-total: 10381124
??++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
### IO statistic on DUT-B ###
??---------------------- Forward statistics for port 0 ?----------------------
RX-packets: 10381124 ??????RX-dropped: 0 ????????????RX-total:?10381124
TX-packets: 10381124 ??????TX-dropped: 0 ????????????TX-total:?10381124
??----------------------------------------------------------------------------
??+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
??RX-packets: 10381124 ??????RX-dropped: 0 ????????????RX-total: 10381124
??TX-packets: 10381124 ??????TX-dropped: 0 ????????????TX-total: 10381124
??++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------
5).?總結
本文基于NXP iMX8M Mini系列處理器簡單演示了?DPDK?軟件和驅(qū)動的部署以及示例應用測試吃衅，按照?NXP?官方說明，?iMX8M Plus平臺?ETH1?應該也支持?DPDK腾誉，只是目前在?toradex_5.15-2.0.x-imx branch?下驅(qū)動支持還存在一定問題徘层，后續(xù)更高版本?branch?可能能夠解決。
參考文檔
NXP i.MX_Linux_Reference_Manual ?Chapter 10 Data Plane Development Kit (DPDK)
