yahoo-eng-team team mailing list archive

[George Shuklin <1658024@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

We've performed upgrade juno->kilo->libery->mitaka (one by one) without
rebooting compute hosts.

After mitaka upgrage we found that some of tenant networks are not
functional. Deeper debug shows that in openvswitch tag value in 'other-
config' field in ovs port description does not match actual tag on the
port. (tag field).

This cause openvswitch-agent to set wrong segmentation_id on irrelevant
host-local tags.

Visual symptom: after restarting neutron-openvswitch-agent connectivity
with given port appears for some time, than disappears. Tcdpump on the
physical interface shows, that traffic coming to host with proper
segmentation_id, but instance's replies are send back with wrong
segmentation_id, which belongs to some random network of the different
tenant.

There are two ways to fix this: 
1. reboot host
2. write tag field to the tag value of the port and restart neutron-openvswitch-agent.

Example of the incorrectly filled port (ovs-vsctl port list):

_uuid               : a5bfb91f-78de-4916-b16a-6ea737cf3b6d
bond_active_slave   : []
bond_downdelay      : 0
bond_fake_iface     : false
bond_mode           : []
bond_updelay        : 0
external_ids        : {}
fake_bridge         : false
interfaces          : [7fb9c7a6-963c-4814-b9a4-a23d1a918843]
lacp                : []
mac                 : []
name                : "tap20802dee-34"
other_config        : {net_uuid="9a1923c8-a07d-487e-a96e-310103acd911", network_type=vlan, physical_network=local, segmentation_id="3035", tag="201"}
qos                 : []
statistics          : {}
status              : {}
tag                 : 302
trunks              : []
vlan_mode           : []


This problems repeated in the few installations of openstack, therefore is not a random fluke.

This script [1] fixes bad tags, but I believe this is a rather serious
issue with openvswitch-agent persistency.


[1] https://gist.github.com/amarao/fba1e766cfa217b0342d0fe066aeedd7


Affected version: mitaka, but I believe it related to previous versions, which was: juno, upgraded to kilo, upgraded to liberty, upgraded to mitaka.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1658024

Title:
  Incorrect tag in other-config for openvsiwtch agent after upgrade to
  mitaka

Status in neutron:
  New

Bug description:
  We've performed upgrade juno->kilo->libery->mitaka (one by one)
  without rebooting compute hosts.

  After mitaka upgrage we found that some of tenant networks are not
  functional. Deeper debug shows that in openvswitch tag value in
  'other-config' field in ovs port description does not match actual tag
  on the port. (tag field).

  This cause openvswitch-agent to set wrong segmentation_id on
  irrelevant host-local tags.

  Visual symptom: after restarting neutron-openvswitch-agent
  connectivity with given port appears for some time, than disappears.
  Tcdpump on the physical interface shows, that traffic coming to host
  with proper segmentation_id, but instance's replies are send back with
  wrong segmentation_id, which belongs to some random network of the
  different tenant.

  There are two ways to fix this: 
  1. reboot host
  2. write tag field to the tag value of the port and restart neutron-openvswitch-agent.

  Example of the incorrectly filled port (ovs-vsctl port list):

  _uuid               : a5bfb91f-78de-4916-b16a-6ea737cf3b6d
  bond_active_slave   : []
  bond_downdelay      : 0
  bond_fake_iface     : false
  bond_mode           : []
  bond_updelay        : 0
  external_ids        : {}
  fake_bridge         : false
  interfaces          : [7fb9c7a6-963c-4814-b9a4-a23d1a918843]
  lacp                : []
  mac                 : []
  name                : "tap20802dee-34"
  other_config        : {net_uuid="9a1923c8-a07d-487e-a96e-310103acd911", network_type=vlan, physical_network=local, segmentation_id="3035", tag="201"}
  qos                 : []
  statistics          : {}
  status              : {}
  tag                 : 302
  trunks              : []
  vlan_mode           : []

  
  This problems repeated in the few installations of openstack, therefore is not a random fluke.

  This script [1] fixes bad tags, but I believe this is a rather serious
  issue with openvswitch-agent persistency.

  
  [1] https://gist.github.com/amarao/fba1e766cfa217b0342d0fe066aeedd7

  
  Affected version: mitaka, but I believe it related to previous versions, which was: juno, upgraded to kilo, upgraded to liberty, upgraded to mitaka.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1658024/+subscriptions