yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #69710
[Bug 1736792] [NEW] DSCP marking QOS policy applied to port not properly updating OVS flow table
Public bug reported:
##########
Openstack Newton
OSA 14.2.4
neutron-server 9.3.2.dev3
OVS firewall_driver = openvswitch
##########
After applying a QOS DSCP-marking policy on a neutron port, the OVS
flow-table on the hosting compute node does not get properly updated
with the required flow to add the marking. The work-around has been to
hard stop the instance, wait until the flows are removed, and re-start
the instance allowing the OVS agent to rebuild the necessary flows.
After the flows are fully rebuilt, the flow rule that marks traffic can
be seen.
neutron qos-policy-list
+--------------------------------------+------------+
| id | name |
+--------------------------------------+------------+
| b7c91afa-c1d1-436a-8543-e64f379d2a4f | dscp-green |
| e86ab2c3-3193-40ce-8301-184be922ee6f | dscp-blue |
+--------------------------------------+------------+
neutron qos-policy-show b7c91afa-c1d1-436a-8543-e64f379d2a4f
+-----------------+-----------------------------------------------------------+
| Field | Value |
+-----------------+-----------------------------------------------------------+
| created_at | 2017-11-21T19:23:28Z |
| description | Green zone |
| id | b7c91afa-c1d1-436a-8543-e64f379d2a4f |
| name | dscp-green |
| project_id | abcdefghilklmnop8368966eb510e105 |
| revision_number | 2 |
| rules | 73bb97ef-33d4-4d9e-934a-e016443648ef (type: dscp_marking) |
| shared | True |
| tenant_id | abcdefghilklmnop8368966eb510e105 |
| updated_at | 2017-11-21T19:23:31Z |
+-----------------+-----------------------------------------------------------+
neutron qos-dscp-marking-rule-show 73bb97ef-33d4-4d9e-934a-e016443648ef b7c91afa-c1d1-436a-8543-e64f379d2a4f
+-----------+--------------------------------------+
| Field | Value |
+-----------+--------------------------------------+
| dscp_mark | 16 |
| id | 73bb97ef-33d4-4d9e-934a-e016443648ef |
+-----------+--------------------------------------+
########################
Neutron port info, *prior* to any QOS policy being applied:
neutron port-show 06c15156-1cd1-4eee-b9a1-bcf379556c99
+-----------------------+----------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | oscomp-ho-c200 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| created_at | 2017-12-06T16:50:09Z |
| description | |
| device_id | 50f90ac8-2e3b-43ee-a1fe-4728fb452382 |
| device_owner | compute:nova |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "3767c511-f2d2-4dc3-a222-123456791011", "ip_address": "10.0.3.10"} |
| id | 06c15156-1cd1-4eee-b9a1-bcf379556c99 |
| mac_address | fa:16:3e:06:b1:8f |
| name | |
| network_id | 55555555-9c52-4658-9ca3-d3715ef54ea6 |
| port_security_enabled | True |
| project_id | 35aac3ee14bd447a8782871ed1cee940 |
| qos_policy_id | |
| revision_number | 9 |
| security_groups | 26711be4-7ae8-4fbb-b097-2405bb2e4f39 |
| status | ACTIVE |
| tenant_id | 35aac3ee14bd447a8782871ed1cee940 |
| updated_at | 2017-12-06T16:50:19Z |
+-----------------------+----------------------------------------------------------------------------------+
Partial flow table off the compute linked to port/instance
...
cookie=0xbfa47c9e78d2597c, duration=208.710s, table=0, n_packets=102, n_bytes=10468, idle_age=3, priority=100,in_port=8 actions=load:0x8->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71)
cookie=0xbfa47c9e78d2597c, duration=208.708s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=130 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=208.707s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=134 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=208.707s, table=71, n_packets=1, n_bytes=78, idle_age=205, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=135 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=136 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=11, n_bytes=462, idle_age=40, priority=95,arp,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=208.705s, table=71, n_packets=2, n_bytes=698, idle_age=205, priority=80,udp,reg5=0x8,in_port=8,tp_src=68,tp_dst=67 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=208.704s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=80,udp6,reg5=0x8,in_port=8,tp_src=546,tp_dst=547 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=208.703s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=70,udp,reg5=0x8,in_port=8,tp_src=67,tp_dst=68 actions=drop
cookie=0xbfa47c9e78d2597c, duration=208.703s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=70,udp6,reg5=0x8,in_port=8,tp_src=547,tp_dst=546 actions=drop
cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=83, n_bytes=8840, idle_age=3, priority=65,ct_state=-trk,ip,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15])
cookie=0xbfa47c9e78d2597c, duration=208.705s, table=71, n_packets=4, n_bytes=300, idle_age=196, priority=65,ct_state=-trk,ipv6,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zone=NXM_NX_REG6[0..15])
cookie=0xbfa47c9e78d2597c, duration=208.702s, table=71, n_packets=1, n_bytes=90, idle_age=205, priority=10,ct_state=-trk,reg5=0x8,in_port=8 actions=drop
...
TCPDump of the physical interface for outgoing traffic to 8.8.8.8 to view any markings:
tcpdump -i bond1 -n -nn -v host 8.8.8.8
16:55:48.913100 IP (tos 0x0, ttl 63, id 39606, offset 0, flags [DF], proto ICMP (1), length 84)
10.0.3.10 > 8.8.8.8: ICMP echo request, id 21505, seq 0, length 64
(note the tos 0x0 stating that there is no DSCP mark applied. This is expected)
########################
I then updated the port with the DSCP marking policy
neutron port-update --qos-policy b7c91afa-c1d1-436a-8543-e64f379d2a4f
06c15156-1cd1-4eee-b9a1-bcf379556c99
+-----------------------+----------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | oscomp-ho-c200 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| created_at | 2017-12-06T16:50:09Z |
| description | |
| device_id | 50f90ac8-2e3b-43ee-a1fe-4728fb452382 |
| device_owner | compute:nova |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "3767c511-f2d2-4dc3-a222-123456791011", "ip_address": "10.0.3.10"} |
| id | 06c15156-1cd1-4eee-b9a1-bcf379556c99 |
| mac_address | fa:16:3e:06:b1:8f |
| name | |
| network_id | 55555555-9c52-4658-9ca3-d3715ef54ea6 |
| port_security_enabled | True |
| project_id | 35aac3ee14bd447a8782871ed1cee940 |
| qos_policy_id | b7c91afa-c1d1-436a-8543-e64f379d2a4f |
| revision_number | 12 |
| security_groups | 26711be4-7ae8-4fbb-b097-2405bb2e4f39 |
| status | ACTIVE |
| tenant_id | 35aac3ee14bd447a8782871ed1cee940 |
| updated_at | 2017-12-06T16:58:03Z |
+-----------------------+----------------------------------------------------------------------------------+
(The qos policy can been seen applied to the port)
OVS agent log files on the compute for the port-update:
2017-12-06 16:58:02.910 21677 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8a8d3edc-f0cd-450f-b77b-d46729fc7bb8 - - - - -] Port 06c15156-1cd1-4eee-b9a1-bcf379556c99 updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': u'b7c91afa-c1d1-436a-8543-e64f379d2a4f', u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'55555555-9c52-4658-9ca3-d3715ef54ea6', u'segmentation_id': 2007, u'device_owner': u'compute:nova', u'physical_network': u'physnet1', u'mac_address': u'fa:16:3e:06:b1:8f', u'device': u'06c15156-1cd1-4eee-b9a1-bcf379556c99', u'port_security_enabled': True, u'port_id': u'06c15156-1cd1-4eee-b9a1-bcf379556c99', u'fixed_ips': [{u'subnet_id': u'3767c511-f2d2-4dc3-a222-123456791011', u'ip_address': u'10.0.3.10'}], u'network_type': u'vlan'}
2017-12-06 16:58:09.322 21677 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8a8d3edc-f0cd-450f-b77b-d46729fc7bb8 - - - - -] Configuration for devices up [u'06c15156-1cd1-4eee-b9a1-bcf379556c99'] and devices down [] completed.
After a few minutes of waiting, there are still no flows to modify
traffic with the DSCP mark:
OVS FLOWS
...
cookie=0xbfa47c9e78d2597c, duration=60.075s, table=0, n_packets=13, n_bytes=878, idle_age=1, priority=100,in_port=8 actions=load:0x8->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71)
cookie=0xbfa47c9e78d2597c, duration=60.073s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=130 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=60.072s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=134 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=60.072s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=135 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=136 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=2, n_bytes=84, idle_age=23, priority=95,arp,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=60.070s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=80,udp,reg5=0x8,in_port=8,tp_src=68,tp_dst=67 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=60.069s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=80,udp6,reg5=0x8,in_port=8,tp_src=546,tp_dst=547 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=60.069s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=70,udp,reg5=0x8,in_port=8,tp_src=67,tp_dst=68 actions=drop
cookie=0xbfa47c9e78d2597c, duration=60.068s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=70,udp6,reg5=0x8,in_port=8,tp_src=547,tp_dst=546 actions=drop
cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=11, n_bytes=794, idle_age=1, priority=65,ct_state=-trk,ip,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15])
cookie=0xbfa47c9e78d2597c, duration=60.070s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=65,ct_state=-trk,ipv6,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zone=NXM_NX_REG6[0..15])
cookie=0xbfa47c9e78d2597c, duration=60.068s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=10,ct_state=-trk,reg5=0x8,in_port=8 actions=drop
...
TCPDump still shows no tos mark:
TCPDUMP
tcpdump -i bond1 -n -nn -v host 8.8.8.8
17:00:37.167559 IP (tos 0x0, ttl 63, id 38836, offset 0, flags [DF], proto ICMP (1), length 84)
10.0.3.10 > 8.8.8.8: ICMP echo request, id 21761, seq 0, length 64
########################
As a workaround, the instance was hard stopped allowing the flows to be
deleted. Then the instance is started and the flows re-created.
nova stop instance_id
(wait a few moments)
nova start instance_id
Here is a new snipit of the flow-table with the "mod_nw_tos" action. (first line below).
OVS FLOWS
...
cookie=0xb6082f15d4334178, duration=447.524s, table=0, n_packets=30, n_bytes=3864, idle_age=3, priority=65535,reg2=0,in_port=10 actions=mod_nw_tos:64,load:0x37->NXM_NX_REG2[0..5],resubmit(,0)
cookie=0xbfa47c9e78d2597c, duration=447.234s, table=0, n_packets=30, n_bytes=3864, idle_age=3, priority=100,in_port=10 actions=load:0xa->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71)
cookie=0xbfa47c9e78d2597c, duration=447.232s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=130 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=447.232s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=134 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=447.231s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=135 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=136 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=5, n_bytes=210, idle_age=15, priority=95,arp,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=447.229s, table=71, n_packets=2, n_bytes=698, idle_age=390, priority=80,udp,reg5=0xa,in_port=10,tp_src=68,tp_dst=67 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=447.228s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=80,udp6,reg5=0xa,in_port=10,tp_src=546,tp_dst=547 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=447.228s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=70,udp,reg5=0xa,in_port=10,tp_src=67,tp_dst=68 actions=drop
cookie=0xbfa47c9e78d2597c, duration=447.227s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=70,udp6,reg5=0xa,in_port=10,tp_src=547,tp_dst=546 actions=drop
cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=20, n_bytes=2726, idle_age=3, priority=65,ct_state=-trk,ip,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15
])
cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=3, n_bytes=230, idle_age=441, priority=65,ct_state=-trk,ipv6,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zo
ne=NXM_NX_REG6[0..15])
cookie=0xbfa47c9e78d2597c, duration=447.227s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=10,ct_state=-trk,reg5=0xa,in_port=10 actions=drop
...
TCPDUMP:
tcpdump -i bond1 -n -nn -v host 8.8.8.8
17:13:37.694875 IP (tos 0x40, ttl 63, id 32155, offset 0, flags [DF], proto ICMP (1), length 84)
10.0.3.10 > 8.8.8.8: ICMP echo request, id 12801, seq 0, length 64
TCPDump show tos 0x40
########################
Interestingly, we do not see the same behavior on OVS environments using
the hybrid firewall driver. We only see it when using the OVS firewall
driver.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1736792
Title:
DSCP marking QOS policy applied to port not properly updating OVS flow
table
Status in neutron:
New
Bug description:
##########
Openstack Newton
OSA 14.2.4
neutron-server 9.3.2.dev3
OVS firewall_driver = openvswitch
##########
After applying a QOS DSCP-marking policy on a neutron port, the OVS
flow-table on the hosting compute node does not get properly updated
with the required flow to add the marking. The work-around has been
to hard stop the instance, wait until the flows are removed, and re-
start the instance allowing the OVS agent to rebuild the necessary
flows. After the flows are fully rebuilt, the flow rule that marks
traffic can be seen.
neutron qos-policy-list
+--------------------------------------+------------+
| id | name |
+--------------------------------------+------------+
| b7c91afa-c1d1-436a-8543-e64f379d2a4f | dscp-green |
| e86ab2c3-3193-40ce-8301-184be922ee6f | dscp-blue |
+--------------------------------------+------------+
neutron qos-policy-show b7c91afa-c1d1-436a-8543-e64f379d2a4f
+-----------------+-----------------------------------------------------------+
| Field | Value |
+-----------------+-----------------------------------------------------------+
| created_at | 2017-11-21T19:23:28Z |
| description | Green zone |
| id | b7c91afa-c1d1-436a-8543-e64f379d2a4f |
| name | dscp-green |
| project_id | abcdefghilklmnop8368966eb510e105 |
| revision_number | 2 |
| rules | 73bb97ef-33d4-4d9e-934a-e016443648ef (type: dscp_marking) |
| shared | True |
| tenant_id | abcdefghilklmnop8368966eb510e105 |
| updated_at | 2017-11-21T19:23:31Z |
+-----------------+-----------------------------------------------------------+
neutron qos-dscp-marking-rule-show 73bb97ef-33d4-4d9e-934a-e016443648ef b7c91afa-c1d1-436a-8543-e64f379d2a4f
+-----------+--------------------------------------+
| Field | Value |
+-----------+--------------------------------------+
| dscp_mark | 16 |
| id | 73bb97ef-33d4-4d9e-934a-e016443648ef |
+-----------+--------------------------------------+
########################
Neutron port info, *prior* to any QOS policy being applied:
neutron port-show 06c15156-1cd1-4eee-b9a1-bcf379556c99
+-----------------------+----------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | oscomp-ho-c200 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| created_at | 2017-12-06T16:50:09Z |
| description | |
| device_id | 50f90ac8-2e3b-43ee-a1fe-4728fb452382 |
| device_owner | compute:nova |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "3767c511-f2d2-4dc3-a222-123456791011", "ip_address": "10.0.3.10"} |
| id | 06c15156-1cd1-4eee-b9a1-bcf379556c99 |
| mac_address | fa:16:3e:06:b1:8f |
| name | |
| network_id | 55555555-9c52-4658-9ca3-d3715ef54ea6 |
| port_security_enabled | True |
| project_id | 35aac3ee14bd447a8782871ed1cee940 |
| qos_policy_id | |
| revision_number | 9 |
| security_groups | 26711be4-7ae8-4fbb-b097-2405bb2e4f39 |
| status | ACTIVE |
| tenant_id | 35aac3ee14bd447a8782871ed1cee940 |
| updated_at | 2017-12-06T16:50:19Z |
+-----------------------+----------------------------------------------------------------------------------+
Partial flow table off the compute linked to port/instance
...
cookie=0xbfa47c9e78d2597c, duration=208.710s, table=0, n_packets=102, n_bytes=10468, idle_age=3, priority=100,in_port=8 actions=load:0x8->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71)
cookie=0xbfa47c9e78d2597c, duration=208.708s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=130 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=208.707s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=134 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=208.707s, table=71, n_packets=1, n_bytes=78, idle_age=205, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=135 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=136 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=11, n_bytes=462, idle_age=40, priority=95,arp,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=208.705s, table=71, n_packets=2, n_bytes=698, idle_age=205, priority=80,udp,reg5=0x8,in_port=8,tp_src=68,tp_dst=67 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=208.704s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=80,udp6,reg5=0x8,in_port=8,tp_src=546,tp_dst=547 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=208.703s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=70,udp,reg5=0x8,in_port=8,tp_src=67,tp_dst=68 actions=drop
cookie=0xbfa47c9e78d2597c, duration=208.703s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=70,udp6,reg5=0x8,in_port=8,tp_src=547,tp_dst=546 actions=drop
cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=83, n_bytes=8840, idle_age=3, priority=65,ct_state=-trk,ip,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15])
cookie=0xbfa47c9e78d2597c, duration=208.705s, table=71, n_packets=4, n_bytes=300, idle_age=196, priority=65,ct_state=-trk,ipv6,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zone=NXM_NX_REG6[0..15])
cookie=0xbfa47c9e78d2597c, duration=208.702s, table=71, n_packets=1, n_bytes=90, idle_age=205, priority=10,ct_state=-trk,reg5=0x8,in_port=8 actions=drop
...
TCPDump of the physical interface for outgoing traffic to 8.8.8.8 to view any markings:
tcpdump -i bond1 -n -nn -v host 8.8.8.8
16:55:48.913100 IP (tos 0x0, ttl 63, id 39606, offset 0, flags [DF], proto ICMP (1), length 84)
10.0.3.10 > 8.8.8.8: ICMP echo request, id 21505, seq 0, length 64
(note the tos 0x0 stating that there is no DSCP mark applied. This is expected)
########################
I then updated the port with the DSCP marking policy
neutron port-update --qos-policy b7c91afa-c1d1-436a-8543-e64f379d2a4f
06c15156-1cd1-4eee-b9a1-bcf379556c99
+-----------------------+----------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | oscomp-ho-c200 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| created_at | 2017-12-06T16:50:09Z |
| description | |
| device_id | 50f90ac8-2e3b-43ee-a1fe-4728fb452382 |
| device_owner | compute:nova |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "3767c511-f2d2-4dc3-a222-123456791011", "ip_address": "10.0.3.10"} |
| id | 06c15156-1cd1-4eee-b9a1-bcf379556c99 |
| mac_address | fa:16:3e:06:b1:8f |
| name | |
| network_id | 55555555-9c52-4658-9ca3-d3715ef54ea6 |
| port_security_enabled | True |
| project_id | 35aac3ee14bd447a8782871ed1cee940 |
| qos_policy_id | b7c91afa-c1d1-436a-8543-e64f379d2a4f |
| revision_number | 12 |
| security_groups | 26711be4-7ae8-4fbb-b097-2405bb2e4f39 |
| status | ACTIVE |
| tenant_id | 35aac3ee14bd447a8782871ed1cee940 |
| updated_at | 2017-12-06T16:58:03Z |
+-----------------------+----------------------------------------------------------------------------------+
(The qos policy can been seen applied to the port)
OVS agent log files on the compute for the port-update:
2017-12-06 16:58:02.910 21677 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8a8d3edc-f0cd-450f-b77b-d46729fc7bb8 - - - - -] Port 06c15156-1cd1-4eee-b9a1-bcf379556c99 updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': u'b7c91afa-c1d1-436a-8543-e64f379d2a4f', u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'55555555-9c52-4658-9ca3-d3715ef54ea6', u'segmentation_id': 2007, u'device_owner': u'compute:nova', u'physical_network': u'physnet1', u'mac_address': u'fa:16:3e:06:b1:8f', u'device': u'06c15156-1cd1-4eee-b9a1-bcf379556c99', u'port_security_enabled': True, u'port_id': u'06c15156-1cd1-4eee-b9a1-bcf379556c99', u'fixed_ips': [{u'subnet_id': u'3767c511-f2d2-4dc3-a222-123456791011', u'ip_address': u'10.0.3.10'}], u'network_type': u'vlan'}
2017-12-06 16:58:09.322 21677 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8a8d3edc-f0cd-450f-b77b-d46729fc7bb8 - - - - -] Configuration for devices up [u'06c15156-1cd1-4eee-b9a1-bcf379556c99'] and devices down [] completed.
After a few minutes of waiting, there are still no flows to modify
traffic with the DSCP mark:
OVS FLOWS
...
cookie=0xbfa47c9e78d2597c, duration=60.075s, table=0, n_packets=13, n_bytes=878, idle_age=1, priority=100,in_port=8 actions=load:0x8->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71)
cookie=0xbfa47c9e78d2597c, duration=60.073s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=130 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=60.072s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=134 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=60.072s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=135 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=136 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=2, n_bytes=84, idle_age=23, priority=95,arp,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=60.070s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=80,udp,reg5=0x8,in_port=8,tp_src=68,tp_dst=67 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=60.069s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=80,udp6,reg5=0x8,in_port=8,tp_src=546,tp_dst=547 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=60.069s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=70,udp,reg5=0x8,in_port=8,tp_src=67,tp_dst=68 actions=drop
cookie=0xbfa47c9e78d2597c, duration=60.068s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=70,udp6,reg5=0x8,in_port=8,tp_src=547,tp_dst=546 actions=drop
cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=11, n_bytes=794, idle_age=1, priority=65,ct_state=-trk,ip,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15])
cookie=0xbfa47c9e78d2597c, duration=60.070s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=65,ct_state=-trk,ipv6,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zone=NXM_NX_REG6[0..15])
cookie=0xbfa47c9e78d2597c, duration=60.068s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=10,ct_state=-trk,reg5=0x8,in_port=8 actions=drop
...
TCPDump still shows no tos mark:
TCPDUMP
tcpdump -i bond1 -n -nn -v host 8.8.8.8
17:00:37.167559 IP (tos 0x0, ttl 63, id 38836, offset 0, flags [DF], proto ICMP (1), length 84)
10.0.3.10 > 8.8.8.8: ICMP echo request, id 21761, seq 0, length 64
########################
As a workaround, the instance was hard stopped allowing the flows to
be deleted. Then the instance is started and the flows re-created.
nova stop instance_id
(wait a few moments)
nova start instance_id
Here is a new snipit of the flow-table with the "mod_nw_tos" action. (first line below).
OVS FLOWS
...
cookie=0xb6082f15d4334178, duration=447.524s, table=0, n_packets=30, n_bytes=3864, idle_age=3, priority=65535,reg2=0,in_port=10 actions=mod_nw_tos:64,load:0x37->NXM_NX_REG2[0..5],resubmit(,0)
cookie=0xbfa47c9e78d2597c, duration=447.234s, table=0, n_packets=30, n_bytes=3864, idle_age=3, priority=100,in_port=10 actions=load:0xa->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71)
cookie=0xbfa47c9e78d2597c, duration=447.232s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=130 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=447.232s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=134 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=447.231s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=135 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=136 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=5, n_bytes=210, idle_age=15, priority=95,arp,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL
cookie=0xbfa47c9e78d2597c, duration=447.229s, table=71, n_packets=2, n_bytes=698, idle_age=390, priority=80,udp,reg5=0xa,in_port=10,tp_src=68,tp_dst=67 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=447.228s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=80,udp6,reg5=0xa,in_port=10,tp_src=546,tp_dst=547 actions=resubmit(,73)
cookie=0xbfa47c9e78d2597c, duration=447.228s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=70,udp,reg5=0xa,in_port=10,tp_src=67,tp_dst=68 actions=drop
cookie=0xbfa47c9e78d2597c, duration=447.227s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=70,udp6,reg5=0xa,in_port=10,tp_src=547,tp_dst=546 actions=drop
cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=20, n_bytes=2726, idle_age=3, priority=65,ct_state=-trk,ip,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15
])
cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=3, n_bytes=230, idle_age=441, priority=65,ct_state=-trk,ipv6,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zo
ne=NXM_NX_REG6[0..15])
cookie=0xbfa47c9e78d2597c, duration=447.227s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=10,ct_state=-trk,reg5=0xa,in_port=10 actions=drop
...
TCPDUMP:
tcpdump -i bond1 -n -nn -v host 8.8.8.8
17:13:37.694875 IP (tos 0x40, ttl 63, id 32155, offset 0, flags [DF], proto ICMP (1), length 84)
10.0.3.10 > 8.8.8.8: ICMP echo request, id 12801, seq 0, length 64
TCPDump show tos 0x40
########################
Interestingly, we do not see the same behavior on OVS environments
using the hybrid firewall driver. We only see it when using the OVS
firewall driver.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1736792/+subscriptions
