OpenShift上的OpenvSwitch入门
2024-10-11 07:18:17
前段时间参加openshift培训，通过产品部门的讲解，刷新了我对OpenShift一些的认识，今天先从最弱的环节网络做一些了解吧。
Openvswitch是openshift sdn的核心组件，进入集群，然后列出某个节点所有的pod
[root@master ~]# oc adm manage-node node1.example.com --list-pods Listing matched pods on node: node1.example.com NAMESPACE NAME READY STATUS RESTARTS AGE
default docker-registry--trrdq / Running 16d
default router--6zjmb / Running 16d
openshift-monitoring alertmanager-main- / Running 16d
openshift-monitoring alertmanager-main- / Running 6d
openshift-monitoring alertmanager-main- / Running 6d
openshift-monitoring cluster-monitoring-operator-df6d9f48d-8lzcw / Running 16d
openshift-monitoring grafana-76cc4f64c-m25c4 / Running 16d
openshift-monitoring kube-state-metrics-8db94b768-tgfgq / Running 16d
openshift-monitoring node-exporter-mjclp / Running 164d
openshift-monitoring prometheus-k8s- / Running 6d
openshift-monitoring prometheus-k8s- / Running 6d
openshift-monitoring prometheus-operator-959fc8dfd-ppc78 / Running 16d
openshift-node sync-7mpsc / Running 164d
openshift-sdn ovs-rzc7j / Running 164d
openshift-sdn sdn-77s8t / Running 164d
Sync Pod
sync pod主要是sync daemonset创建，主要负责监控/etc/sysconfig/atomic-openshift-node的变化， 主要是观察BOOTSTRAP_CONFIG_NAME的设置.
BOOTSTRAP_CONFIG_NAME 是openshift-ansible 安装的，是一个基于node configuration group的ConfigMap类型。openshift 缺省有如下node Configuration Group
node-config-master
node-config-infra
node-config-compute
node-config-all-in-one
node-config-master-infra
Sync Pod 转换configmap的数据到kubelet的配置，并为节点生成/etc/origin/node/node-config.yaml ，如果文件配置有变化，kubelet会重启。
0. OVS的简介
OVS Pod
ovs Pod主要基于Openvswitch为Openshift提供一个容器网络.它是OpenvSwitch的一个容器化实现，核心组件如下：
核心组件介绍如下
- OVS-VSwitchd
ovs-vswitchd守护进程是OVS的核心部件，它和datapath内核模块一起实现OVS基于流的数据交换。作为核心组件，它使用openflow协议与上层OpenFlow控制器通信，使用OVSDB协议与ovsdb-server通信，使用netlink和datapath内核模块通信。ovs-vswitchd在启动时会读取ovsdb-server中配置信息，然后配置内核中的datapaths和所有OVS switches，当ovsdb中的配置信息改变时(例如使用ovs-vsctl工具)，ovs-vswitchd也会自动更新其配置以保持与数据库同步
进入到ovs pod,可以看到运行的核心进程
root : ? :: ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
ovsdb-server
ovsdb-server是OVS轻量级的数据库服务，用于整个OVS的配置信息，包括接口/交换内容/VLAN等，OVS主进程ovs-vswitchd根据数据库中的配置信息工作，下面是ovsdb-server进程详细信息
sh-4.2# ps -ef | grep ovsdb-server
root : ? :: /bin/bash -c #!/bin/bash set -euo pipefail # if another process is listening on the cni-server socket, wait until it exits trap 'kill $(jobs -p); exit 0' TERM retries= while true; do if /usr/share/openvswitch/scripts/ovs-ctl status &>/dev/null; then echo "warning: Another process is currently managing OVS, waiting 15s ..." >& sleep & wait (( retries += )) else break fi if [[ "${retries}" -gt ]]; then echo "error: Another process is currently managing OVS, exiting" >& exit fi done # launch OVS function quit { /usr/share/openvswitch/scripts/ovs-ctl stop exit } trap quit SIGTERM /usr/share/openvswitch/scripts/ovs-ctl start --no-ovs-vswitchd --system-id=random # Restrict the number of pthreads ovs-vswitchd creates to reduce the # amount of RSS it uses on hosts with many cores # https://bugzilla.redhat.com/show_bug.cgi?id=1571379 # https://bugzilla.redhat.com/show_bug.cgi?id=1572797 if [[ `nproc` -gt 12 ]]; then ovs-vsctl --no-wait set Open_vSwitch . other_config:n-revalidator-threads=4 ovs-vsctl --no-wait set Open_vSwitch . other_config:n-handler-threads=10 fi /usr/share/openvswitch/scripts/ovs-ctl start --no-ovsdb-server --system-id=random tail --follow=name /var/log/openvswitch/ovs-vswitchd.log /var/log/openvswitch/ovsdb-server.log & sleep 20 while true; do if ! /usr/share/openvswitch/scripts/ovs-ctl status &>/dev/null; then echo "OVS seems to have crashed, exiting" quit fi sleep 15 done
root : ? :: ovsdb-server: monitoring pid (healthy)
root : ? :: ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
root : ? :: tail --follow=name /var/log/openvswitch/ovs-vswitchd.log /var/log/openvswitch/ovsdb-server.log
root : ? :: grep ovsdb-server
OpenFlow
OpenFlow是开源的用于管理交换机流表的协议，OpenFlow在OVS中的地位可以参考上面架构图，它是Controller和ovs-vswitched间的通信协议。需要注意的是，OpenFlow是一个独立的完整的流表协议，不依赖于OVS，OVS只是支持OpenFlow协议，有了支持，我们可以使用OpenFlow控制器来管理OVS中的流表，OpenFlow不仅仅支持虚拟交换机，某些硬件交换机也支持OpenFlow协议。
Controller
Controller指OpenFlow控制器。OpenFlow控制器可以通过OpenFlow协议连接到任何支持OpenFlow的交换机，比如OVS。控制器通过向交换机下发流表规则来控制数据流向。除了可以通过OpenFlow控制器配置OVS中flows，也可以使用OVS提供的ovs-ofctl命令通过OpenFlow协议去连接OVS，从而配置flows，命令也能够对OVS的运行状况进行动态监控。
Kernel Datapath
datapath是一个Linux内核模块，它负责执行数据交换。
1.网络架构
下面这个图比较直观。
找个环境先看看
[root@node1 ~]# ip addr show
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default qlen
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
valid_lft forever preferred_lft forever
inet6 ::/ scope host
valid_lft forever preferred_lft forever
: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP group default qlen
link/ether :::dc::1a brd ff:ff:ff:ff:ff:ff
inet 192.168.56.104/ brd 192.168.56.255 scope global noprefixroute enp0s3
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fedc:991a/ scope link tentative dadfailed
valid_lft forever preferred_lft forever
: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::1e::6a: brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/ scope global docker0
valid_lft forever preferred_lft forever
: ovs-system: <BROADCAST,MULTICAST> mtu qdisc noop state DOWN group default qlen
link/ether :ea::f3:: brd ff:ff:ff:ff:ff:ff
: br0: <BROADCAST,MULTICAST> mtu qdisc noop state DOWN group default qlen
link/ether 8e:be:::b7: brd ff:ff:ff:ff:ff:ff
: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UNKNOWN group default qlen
link/ether :e0:7e::e9:5b brd ff:ff:ff:ff:ff:ff
inet6 fe80::54e0:7eff:fe45:e95b/ scope link
valid_lft forever preferred_lft forever
: tun0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default qlen
link/ether 1a::ef:7d::2a brd ff:ff:ff:ff:ff:ff
inet 10.130.0.1/ brd 10.130.1.255 scope global tun0
valid_lft forever preferred_lft forever
inet6 fe80:::efff:fe7d:842a/ scope link
valid_lft forever preferred_lft forever
: veth4da99f3a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UP group default
link/ether 4e:7c::3b:db:3c brd ff:ff:ff:ff:ff:ff link-netnsid
inet6 fe80::4c7c:96ff:fe3b:db3c/ scope link
valid_lft forever preferred_lft forever
: veth4596fc96@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UP group default
link/ether 2a::b7:1b:e0: brd ff:ff:ff:ff:ff:ff link-netnsid
inet6 fe80:::b7ff:fe1b:e081/ scope link
valid_lft forever preferred_lft forever
: veth04caa6b2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UP group default
link/ether 3a:5b:9a::9c:f4 brd ff:ff:ff:ff:ff:ff link-netnsid
inet6 fe80::385b:9aff:fe62:9cf4/ scope link
valid_lft forever preferred_lft forever
: veth14f14b18@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UP group default
link/ether ca:d2::::be brd ff:ff:ff:ff:ff:ff link-netnsid
inet6 fe80::c8d2:96ff:fe48:84be/ scope link
valid_lft forever preferred_lft forever
: veth31713a78@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UP group default
link/ether ce:c3:8f:3e:b9: brd ff:ff:ff:ff:ff:ff link-netnsid
inet6 fe80::ccc3:8fff:fe3e:b941/ scope link
valid_lft forever preferred_lft forever
: veth9ff2f96a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UP group default
link/ether :f8::d2::7a brd ff:ff:ff:ff:ff:ff link-netnsid
inet6 fe80::14f8:11ff:fed2:287a/ scope link
valid_lft forever preferred_lft forever
: veth86a4a302@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UP group default
link/ether ::ab:7a:: brd ff:ff:ff:ff:ff:ff link-netnsid
inet6 fe80:::abff:fe7a:/ scope link
valid_lft forever preferred_lft forever
: vethb4141622@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UP group default
link/ether :d1:df::5c:d3 brd ff:ff:ff:ff:ff:ff link-netnsid
inet6 fe80::44d1:dfff:fe02:5cd3/ scope link
valid_lft forever preferred_lft forever
: vethae772509@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UP group default
link/ether c6::af::f0:b7 brd ff:ff:ff:ff:ff:ff link-netnsid
inet6 fe80::c412:afff:fe03:f0b7/ scope link
valid_lft forever preferred_lft forever
: veth4fbfae38@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue master ovs-system state UP group default
link/ether :f3:e8::9b:e9 brd ff:ff:ff:ff:ff:ff link-netnsid
inet6 fe80::80f3:e8ff:fe87:9be9/ scope link
valid_lft forever preferred_lft forever
可以看到一堆的interface, 在一个节点上，OpenShift SDN往往会生成如下的类型的接口(interface)
br0: OVS的网桥设备 The OVS bridge device that containers will be attached to. OpenShift SDN also configures a set of non-subnet-specific flow rules on this bridge.tun0: 访问外部网络的网关 An OVS internal port (port 2 onbr0). This gets assigned the cluster subnet gateway address, and is used for external network access. OpenShift SDN configuresnetfilterand routing rules to enable access from the cluster subnet to the external network via NAT.vxlan_sys_4789: 访问其他节点Pod的设备 The OVS VXLAN device (port 1 onbr0), which provides access to containers on remote nodes. Referred to asvxlan0in the OVS rules.vethX(in the main netns): 和Pod关联的虚拟网卡 A Linux virtual ethernet peer ofeth0in the Docker netns. It will be attached to the OVS bridge on one of the other ports.
这个说明可以结合上面那个图来看，就很清晰了。
基于brctl命令是看不到ovs的网桥的，只能看到 docker0
[root@node1 ~]# brctl show
bridge name bridge id STP enabled interfaces
docker0 .02421e326a69 no
如果要看到br0网桥，并且了解附在上面的ip信息，需要安装openvswitch.
2.安装OpenvSwitch
基于
https://access.redhat.com/solutions/3710321
因为openvswitch 在3.10以上版本转为static pod实现，所以在3.10以上版本在节点不预装openvswitch,如果需要安装参照一下步骤。当然也可以直接在ovs pod里面进行查看。
# subscription-manager register
# subscription-manager list --available
# subscription-manager attach --pool=<pool_id>
subscription-manager repos --enable=rhel--server-extras-rpms
subscription-manager repos --enable=rhel--server-optional-rpms yum install -y openvswitch
基于命令查看网桥
[root@node1 network-scripts]# ovs-vsctl list-br
br0
查看网桥上的端口
[root@node1 network-scripts]# ovs-ofctl -O OpenFlow13 dump-ports-desc br0
OFPST_PORT_DESC reply (OF1.) (xid=0x2):
(vxlan0): addr:a2::b9:7d:a6:c3
config:
state: LIVE
speed: Mbps now, Mbps max
(tun0): addr:de::d6:e7:a9:
config:
state: LIVE
speed: Mbps now, Mbps max
(veth04fb5821): addr::::fa:e0:9f
config:
state: LIVE
current: 10GB-FD COPPER
speed: Mbps now, Mbps max
(vethbdebc4f7): addr:d6:f8:::3e:da
config:
state: LIVE
current: 10GB-FD COPPER
speed: Mbps now, Mbps max
(veth9fd3926f): addr:5e:::c8::9e
config:
state: LIVE
current: 10GB-FD COPPER
speed: Mbps now, Mbps max
(veth3b466fa9): addr:ee:3f:1b:cb:cf:9b
config:
state: LIVE
current: 10GB-FD COPPER
speed: Mbps now, Mbps max
(veth866c42b5): addr:be:7e:e6:d2:2f:f1
config:
state: LIVE
current: 10GB-FD COPPER
speed: Mbps now, Mbps max
(veth2446bc66): addr::5b:fe:::
config:
state: LIVE
current: 10GB-FD COPPER
speed: Mbps now, Mbps max
(veth1afcb012): addr::3b:de:9d::8b
config:
state: LIVE
current: 10GB-FD COPPER
speed: Mbps now, Mbps max
(vethd31bb8c7): addr::8f:::ba:
config:
state: LIVE
current: 10GB-FD COPPER
speed: Mbps now, Mbps max
(vethe02a7907): addr::c4:af:c3:c8:
config:
state: LIVE
current: 10GB-FD COPPER
speed: Mbps now, Mbps max
(veth06b13117): addr:0a:::5e::d7
config:
state: LIVE
current: 10GB-FD COPPER
speed: Mbps now, Mbps max
LOCAL(br0): addr:b2:a3:b4::9d:4e
config: PORT_DOWN
state: LINK_DOWN
speed: Mbps now, Mbps max
查看所有信息
[root@node1 ~]# ovs-vsctl show
c2bad35f---aa1c-b255d0ee7e60
Bridge "br0"
fail_mode: secure
Port "tun0"
Interface "tun0"
type: internal
Port "vethfff599a6"
Interface "vethfff599a6"
Port "br0"
Interface "br0"
type: internal
Port "veth38efc56f"
Interface "veth38efc56f"
Port "veth80b6c074"
Interface "veth80b6c074"
Port "veth4cdb026d"
Interface "veth4cdb026d"
Port "veth42f411d0"
Interface "veth42f411d0"
Port "vxlan0"
Interface "vxlan0"
type: vxlan
options: {dst_port="", key=flow, remote_ip=flow}
Port "veth5bf0c012"
Interface "veth5bf0c012"
Port "vethb80cca24"
Interface "vethb80cca24"
Port "veth2335064f"
Interface "veth2335064f"
Port "vethf857e799"
Interface "vethf857e799"
Port "veth325c8496"
Interface "veth325c8496"
ovs_version: "2.9.0"
查看流量
[root@node1 network-scripts]# ovs-ofctl -O OpenFlow13 dump-flows br0
OFPST_FLOW reply (OF1.) (xid=0x2):
cookie=0x0, duration=.192s, table=, n_packets=, n_bytes=, priority=,ip,in_port=,nw_dst=224.0.0.0/ actions=drop
cookie=0x0, duration=.192s, table=, n_packets=, n_bytes=, priority=,arp,in_port=,arp_spa=10.128.0.0/,arp_tpa=10.130.0.0/ actions=move:NXM_NX_TUN_ID[..]->NXM_NX_REG0[],goto_table:
cookie=0x0, duration=.192s, table=, n_packets=, n_bytes=, priority=,ip,in_port=,nw_src=10.128.0.0/ actions=move:NXM_NX_TUN_ID[..]->NXM_NX_REG0[],goto_table:
cookie=0x0, duration=.192s, table=, n_packets=, n_bytes=, priority=,ip,in_port=,nw_dst=10.128.0.0/ actions=move:NXM_NX_TUN_ID[..]->NXM_NX_REG0[],goto_table:
cookie=0x0, duration=.192s, table=, n_packets=, n_bytes=, priority=,arp,in_port=,arp_spa=10.130.0.1,arp_tpa=10.128.0.0/ actions=goto_table:
开了个头，先这样，更多的网络诊断，可以参考
https://docs.openshift.com/container-platform/3.11/admin_guide/sdn_troubleshooting.html
openvswitch材料，参考
https://opengers.github.io/openstack/openstack-base-use-openvswitch/
最新文章
- Ubuntu apt-get "Hash Sum mismatch" 问题解决方法
- ios AFNetworking 3.0 报错 ： *** Assertion failure in -[AFHTTPRequestSerializer requestWithMethod:URLString:parameters:error:],
- CentOS 6.5安装配置LNMP服务器（Nginx+PHP+MySQL）
- 超微 X9DRL-iF 服务器主板简介 BIOS相关图解
- java项目中读取src目录下的文件
- HTML语义化标签（一）
- 【剑指offer】面试题26：复杂链表的复制
- UVA - 11324 The Largest Clique 强连通缩点+记忆化dp
- [置顶] 如何高效使用和管理Bitmap--图片缓存管理模块的设计与实现
- SVN：cannot map the project with svn provider解决办法
- 第三方登录 ----QQ登录
- Java关于e.printStackTrace()介绍
- WIP完工入库及完工退回的几个重要问题
- mysql很有用的语句
- hystrix项目实战
- 自学Zabbix6.1 Event acknowledgment 事件确认
- poj 2175 费用流消圈
- Windows下安装pymssql
- nginx 常见错误释义
- Spring事务内方法调用自身事务 增强的三种方式