[dpdk] 读官方文档(1)

前提：已读了这本书<<深入浅出dpdk（朱清河等著）>>。

目标：读官方文档，同时跟着文档进行安装编译等工作。

http://dpdk.org/doc/guides/index.html

环境：thinkpadT450 + archlinux + kvm + CentOS 7（逐层自下而上的关系）

KVM参数：

sudo qemu-system-x86_64 -nographic -enable-kvm -numa node -numa node -drive file=disk.img,if=virtio \

　　-cdrom /home/tong/ISO/CentOS--x86_64-DVD-.iso -boot order=c -m 2G \

　　-net nic,model=virtio,macaddr='00:00:00:00:00:03' -net tap,ifname=tap0

当前dpdk版本：dpdk-16.07

前提交代完，接下来是正文。内容会是一种笔记的形式，我遇见问题，就会在下边写下来。

一编译：

make config T=x86_64-native-linuxapp-gcc

sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config

make

问题一：编译错误

In file included from /root/dpdk-16.07/lib/librte_eal/linuxapp/eal/eal_pci.c:::

/root/dpdk-16.07/build/include/rte_memcpy.h::: error: incompatible type for argument  of ‘_mm_storeu_si128’

  MOVEUNALIGNED_LEFT47(dst, src, n, srcofs);

  ^

In file included from /root/dpdk-16.07/build/include/rte_common.h::,

                 from /root/dpdk-16.07/build/include/rte_log.h:,

                 from /root/dpdk-16.07/lib/librte_eal/linuxapp/eal/eal_pci.c::

/usr/lib/gcc/x86_64-redhat-linux/4.8./include/emmintrin.h::: note: expected ‘__m128i’ but argument is of type ‘int’

 _mm_storeu_si128 (__m128i *__P, __m128i __B)

咋解决：查看了这个头文件emmintrin.h 内容是这样的，也就是说这是个CPU平台相关的问题，所以很可能需要调整kvm的cpu参数，或者dpdk的编译参数，从这两个方面入手：

/* The Intel API is flexible enough that we must allow aliasing with other

   vector types, and their scalar components.  */

typedef long long __m128i __attribute__ ((__vector_size__ (), __may_alias__));

typedef double __m128d __attribute__ ((__vector_size__ (), __may_alias__));

dpdk的编译参数并没有神马好改的，你重要指定一个CPU平台给它，所以我必须让qemu模拟intel或者是虚拟机直接认得本地CPU，然而qemu的手册并不清晰，又没有时间系统的学习qemu，只能试了。（qemu kvm也需要系统学习！）

尝试1：不给qemu CPU参数：

 qemu-system-x86_64 -nographic -enable-kvm -m 2G -drive file=disk.img,if=virtio \
      -net nic,model=virtio,macaddr='00:00:00:00:00:03' -net tap,ifname=tap0

进入系统后查看cpuinfo

[root@dpdk dpdk-16.07]# cat /proc/cpuinfo

processor       :

vendor_id       : GenuineIntel

cpu family      :

model           :

model name      : QEMU Virtual CPU version 2.5+

stepping        :

......

编译报同样的错，毫无悬念。

尝试2：指定CPU类型，模拟至强E3

qemu-system-x86_64 -nographic -enable-kvm -m 2G -cpu IvyBridge -drive file=disk.img,if=virtio -net nic,model=virtio,macaddr='00:00:00:00:00:03' -net tap,ifname=tap0

编译成功：

[root@dpdk dpdk-16.07]# cat /proc/cpuinfo

processor       :

vendor_id       : GenuineIntel

cpu family      :

model           :

model name      : Intel Xeon E3-12xx v2 (Ivy Bridge)

stepping        :

microcode       : 0x1

cpu MHz         : 2394.456

cache size      :  KB

... ...

编译错误二：

  SYMLINK-FILE include/exec-env/rte_dom0_common.h

  INSTALL-LIB librte_eal.a

== Build lib/librte_eal/linuxapp/igb_uio

make: *** /lib/modules/3.10.-.el7.x86_64/build: No such file or directory.  Stop.

make[]: *** [igb_uio.ko] Error

make[]: *** [igb_uio] Error

make[]: *** [linuxapp] Error

make[]: *** [librte_eal] Error

make[]: *** [lib] Error

make: *** [all] Error

解决：人家文档写的清楚需要内核开发包 “kernel - devel.x86_64”

yum install kernel-devel

ln -fs /usr/src/kernels/3.10.-327.36..el7.x86_64/ /lib/modules/3.10.-.el7.x86_64/build

问题三：因为enable了PCAP，没错就是前边那个sed，鬼知道它是干嘛的，反正要装 libpcap-devel.

编译通过，好棒！: )

二，调整硬件架构

调整虚拟机为numa架构，并设置hugepage。

# numa参数只调整结构，并不申请资源，所以还是需要m和smp两个参数来进行资源申请。
sudo qemu-system-x86_64 -nographic -vnc 127.0.0.1:0 -enable-kvm \

        -m 2G -cpu Nehalem \                  #Nehalem就是i7，我的物理CPU是i7，模拟至强的时候，warning有一些特性不支持，为了防止后续出现诡异的错误，就模拟了i7.

        -smp cores=,threads=,sockets= \    #两颗CPU，两个核心，超线程

        -numa node,mem=1G,cpus=-,nodeid= \

        -numa node,mem=1G,cpus=-,nodeid= \

        -drive file=disk.img,if=virtio \

        -net nic,model=virtio,macaddr='00:00:00:00:00:03' \

        -net tap,ifname=tap0 &

然后在虚拟机里的效果就是这样的：

[root@dpdk ~]# cat /proc/cpuinfo

processor       :

vendor_id       : GenuineIntel

cpu family      :

model   :

model name      : Intel Core i7 9xx (Nehalem Class Core i7)

stepping        :

microcode       : 0x1

cpu MHz         : 2394.456

cache size      :  KB

physical id     :

siblings        :

core id         :

cpu cores       :

apicid          :

initial apicid  :

fpu     : yes

fpu_exception   : yes

cpuid level     :

wp      : yes

flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm

bogomips        : 4788.91

clflush size    :

cache_alignment :

address sizes   :  bits physical,  bits virtual

power management:

processor       :

vendor_id       : GenuineIntel

cpu family      :

model   :

model name      : Intel Core i7 9xx (Nehalem Class Core i7)

stepping        :

microcode       : 0x1

cpu MHz         : 2394.456

cache size      :  KB

physical id     :

siblings        :

core id         :

cpu cores       :

apicid          :

initial apicid  :

fpu     : yes

fpu_exception   : yes

cpuid level     :

wp      : yes

flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm

bogomips        : 4788.91

clflush size    :

cache_alignment :

address sizes   :  bits physical,  bits virtual

power management:

processor       :

vendor_id       : GenuineIntel

cpu family      :

model   :

model name      : Intel Core i7 9xx (Nehalem Class Core i7)

stepping        :

microcode       : 0x1

cpu MHz         : 2394.456

cache size      :  KB

physical id     :

siblings        :

core id         :

cpu cores       :

apicid          :

initial apicid  :

fpu     : yes

fpu_exception   : yes

cpuid level     :

wp      : yes

flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm

bogomips        : 4788.91

clflush size    :

cache_alignment :

address sizes   :  bits physical,  bits virtual

power management:

processor       :

vendor_id       : GenuineIntel

cpu family      :

model   :

model name      : Intel Core i7 9xx (Nehalem Class Core i7)

stepping        :

microcode       : 0x1

cpu MHz         : 2394.456

cache size      :  KB

physical id     :

siblings        :

core id         :

cpu cores       :

apicid          :

initial apicid  :

fpu     : yes

fpu_exception   : yes

cpuid level     :

wp      : yes

flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm

bogomips        : 4788.91

clflush size    :

cache_alignment :

address sizes   :  bits physical,  bits virtual

power management:

processor       :

vendor_id       : GenuineIntel

cpu family      :

model   :

model name      : Intel Core i7 9xx (Nehalem Class Core i7)

stepping        :

microcode       : 0x1

cpu MHz         : 2394.456

cache size      :  KB

physical id     :

siblings        :

core id         :

cpu cores       :

apicid          :

initial apicid  :

fpu     : yes

fpu_exception   : yes

cpuid level     :

wp      : yes

flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm

bogomips        : 4788.91

clflush size    :

cache_alignment :

address sizes   :  bits physical,  bits virtual

power management:

processor       :

vendor_id       : GenuineIntel

cpu family      :

model   :

model name      : Intel Core i7 9xx (Nehalem Class Core i7)

stepping        :

microcode       : 0x1

cpu MHz         : 2394.456

cache size      :  KB

physical id     :

siblings        :

core id         :

cpu cores       :

apicid          :

initial apicid  :

fpu     : yes

fpu_exception   : yes

cpuid level     :

wp      : yes

flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm

bogomips        : 4788.91

clflush size    :

cache_alignment :

address sizes   :  bits physical,  bits virtual

power management:

processor       :

vendor_id       : GenuineIntel

cpu family      :

model   :

model name      : Intel Core i7 9xx (Nehalem Class Core i7)

stepping        :

microcode       : 0x1

cpu MHz         : 2394.456

cache size      :  KB

physical id     :

siblings        :

core id         :

cpu cores       :

apicid          :

initial apicid  :

fpu     : yes

fpu_exception   : yes

cpuid level     :

wp      : yes

flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm

bogomips        : 4788.91

clflush size    :

cache_alignment :

address sizes   :  bits physical,  bits virtual

power management:

processor       :

vendor_id       : GenuineIntel

cpu family      :

model   :

model name      : Intel Core i7 9xx (Nehalem Class Core i7)

stepping        :

microcode       : 0x1

cpu MHz         : 2394.456

cache size      :  KB

physical id     :

siblings        :

core id         :

cpu cores       :

apicid          :

initial apicid  :

fpu     : yes

fpu_exception   : yes

cpuid level     :

wp      : yes

flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm

bogomips        : 4788.91

clflush size    :

cache_alignment :

address sizes   :  bits physical,  bits virtual

power management:

cat /proc/cpuinfo

[root@dpdk ~]# numastat

                           node0           node1

numa_hit

numa_miss

numa_foreign

interleave_hit

local_node

other_node

numastat

大页：

查看cpuinfo： If pse exists, 2M hugepages are supported; if pdpe1gb exists, 1G hugepages are supported. 所以我只支持2M，而且我也没有那么多内存。但是我的物理cpu是支持1G的，这让我感觉很牛逼，我是说host主机。

通过设置内核参数：测试发现，保留的大页会平均分给两个node。

[root@dpdk ~]# cat /boot/grub2/grub.cfg|grep huge

        linux16 /vmlinuz-3.10.-.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF- hugepages=

[root@dpdk ~]# cat /proc/meminfo |grep Huge

AnonHugePages:       kB

HugePages_Total:

HugePages_Free:

HugePages_Rsvd:

HugePages_Surp:

Hugepagesize:        kB

[root@dpdk ~]# cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages 

[root@dpdk ~]# cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages

挂载大页到启动项里，（干这个事的时候出现了插曲，搞了一下nbd和lvm的挂载，真爱粉们可以去另一篇里边追剧哦），截止到目前完全不知道为啥要这样做。

[root@dpdk ~]# cat /etc/fstab

... ...

# custom

nodev /mnt/huge hugetlbfs defaults

[root@dpdk ~]#

反正，测试程序 testpmd能跑起来了，算告一段落，太长了，我要切！

巴特西

[dpdk] 读官方文档(1)

最新文章

热门文章