yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1379663] Re: After upgrading - ovs-vswitchd cannot add existing ports

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "Markus Zoeller \(markus_z\)" <mzoeller@xxxxxxxxxxxxxxxxxx>
Date: Tue, 05 Jul 2016 09:53:44 -0000
Reply-to: Bug 1379663 <1379663@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON).
  Valid example: CONFIRMED FOR: LIBERTY


** Changed in: nova
       Status: Confirmed => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1379663

Title:
  After upgrading - ovs-vswitchd cannot add existing ports

Status in neutron:
  Incomplete
Status in OpenStack Compute (nova):
  Expired

Bug description:
  Hi there,

  After upgrading (stop all services, yum upgrade, db sync) from older
  Icehouse build to latest Icehouse build, my compute node (specifically
  openstack-nova-compute) cannot be started. I deployed a number of
  instances before upgrade, and after upgrading openstack-nova-compute
  refuses to start up. The logs seem to point to some issue with ovs-
  vswitch unable to bind ports of the existing instances.

  All other services at controller and network nodes seem to be running
  fine. And before upgrading, everything was working fine.

  # rpm -qa | grep openstack-nova
  openstack-nova-compute-2014.1.2-1.el6.noarch
  openstack-nova-common-2014.1.2-1.el6.noarch

  At compute.log:

  2014-10-10 14:37:39.372 24897 ERROR nova.openstack.common.threadgroup [-] Unexpected vif_type=binding_failed
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     x.wait()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 173, in wait
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 121, in wait
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 293, in switch
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 212, in main
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 486, in run_service
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     service.start()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/service.py", line 163, in start
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     self.manager.init_host()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1044, in init_host
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     self._init_instance(context, instance)
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 902, in _init_instance
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     self.driver.plug_vifs(instance, net_info)
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 860, in plug_vifs
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     self.vif_driver.plug(instance, vif)
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/vif.py", line 616, in plug
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     _("Unexpected vif_type=%s") % vif_type)
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup NovaException: Unexpected vif_type=binding_failed
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup

  
  At ovs-vswitchd.log:

  2014-10-10T06:37:38Z|00377|dpif|WARN|Dropped 9 log messages in last 17953 seconds (most recently, 17952 seconds ago) due to excessive rate
  2014-10-10T06:37:38Z|00378|dpif|WARN|system@ovs-system: port_del failed (Invalid argument)
  2014-10-10T06:37:38Z|00379|netdev_linux|WARN|Dropped 8 log messages in last 17953 seconds (most recently, 17952 seconds ago) due to excessive rate
  2014-10-10T06:37:38Z|00380|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:38Z|00381|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device
  2014-10-10T06:37:38Z|00382|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:38Z|00383|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device
  2014-10-10T06:37:38Z|00384|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:38Z|00385|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device
  2014-10-10T06:37:44Z|00386|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:44Z|00387|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device
  2014-10-10T06:37:44Z|00388|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:44Z|00389|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device
  2014-10-10T06:37:44Z|00390|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:44Z|00391|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device
  2014-10-10T06:37:44Z|00392|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:44Z|00393|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device
  2014-10-10T06:37:44Z|00394|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:44Z|00395|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device
  2014-10-10T06:37:44Z|00396|bridge|INFO|bridge br-tun: using datapath ID 00003666e785084d
  2014-10-10T06:37:44Z|00397|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt"
  2014-10-10T06:37:44Z|00398|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:44Z|00399|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device
  2014-10-10T06:37:44Z|00400|bridge|INFO|bridge br-tun: added interface br-tun on port 65534
  2014-10-10T06:37:44Z|00401|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:44Z|00402|bridge|INFO|bridge br-int: added interface patch-tun on port 1
  2014-10-10T06:37:44Z|00403|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:44Z|00404|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:44Z|00405|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:44Z|00406|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:44Z|00407|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:44Z|00408|bridge|INFO|bridge br-tun: added interface patch-int on port 1
  2014-10-10T06:37:45Z|00409|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:45Z|00410|dpif|WARN|Dropped 6 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate
  2014-10-10T06:37:45Z|00411|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device
  2014-10-10T06:37:45Z|00412|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:45Z|00413|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:45Z|00414|bridge|INFO|bridge br-tun: added interface gre-c0a80547 on port 2
  2014-10-10T06:37:48Z|00415|ofproto|INFO|br-int: 3 flow_mods 10 s ago (2 adds, 1 deletes)
  2014-10-10T06:37:54Z|00416|ofproto|INFO|br-tun: 13 flow_mods in the 1 s starting 10 s ago (11 adds, 1 deletes, 1 modifications)

  But then these ports are already running under br-int -

  # ovs-vsctl show

  8fa6592e-b814-4c90-8ce5-8ee2f2e61d39
      Bridge br-int
          fail_mode: secure
          Port "qvoeda2db74-94"
              tag: 4095
              Interface "qvoeda2db74-94"
          Port patch-tun
              Interface patch-tun
                  type: patch
                  options: {peer=patch-int}
          Port "qvo2beb3ffb-6d"
              tag: 1
              Interface "qvo2beb3ffb-6d"
          Port "qvo1a5fea4a-6e"
              tag: 1
              Interface "qvo1a5fea4a-6e"
          Port "qvo684ea2ea-63"
              tag: 1
              Interface "qvo684ea2ea-63"
          Port "qvoeae48b5f-9b"
              tag: 1
              Interface "qvoeae48b5f-9b"
          Port br-int
              Interface br-int
                  type: internal
      Bridge br-tun
          Port patch-int
              Interface patch-int
                  type: patch
                  options: {peer=patch-tun}
          Port br-tun
              Interface br-tun
                  type: internal
          Port "gre-c0a80547"
              Interface "gre-c0a80547"
                  type: gre
                  options: {in_key=flow, local_ip="192.168.5.72", out_key=flow, remote_ip="192.168.5.71"}
      ovs_version: "1.11.0"

  Regards,
  Boon Lee

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1379663/+subscriptions
References

[Bug 1379663] [NEW] After upgrading - ovs-vswitchd cannot add existing ports
From: Boon Lee, 2014-10-10