yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1379663] Re: After upgrading - ovs-vswitchd cannot add existing ports

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Rodolfo Alonso <1379663@xxxxxxxxxxxxxxxxxx>
Date: Wed, 30 Nov 2022 10:06:38 -0000
Reply-to: Bug 1379663 <1379663@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
** Changed in: neutron
       Status: Incomplete => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1379663

Title:
  After upgrading - ovs-vswitchd cannot add existing ports

Status in neutron:
  Won't Fix
Status in OpenStack Compute (nova):
  Expired

Bug description:
  Hi there,

  After upgrading (stop all services, yum upgrade, db sync) from older
  Icehouse build to latest Icehouse build, my compute node (specifically
  openstack-nova-compute) cannot be started. I deployed a number of
  instances before upgrade, and after upgrading openstack-nova-compute
  refuses to start up. The logs seem to point to some issue with ovs-
  vswitch unable to bind ports of the existing instances.

  All other services at controller and network nodes seem to be running
  fine. And before upgrading, everything was working fine.

  # rpm -qa | grep openstack-nova
  openstack-nova-compute-2014.1.2-1.el6.noarch
  openstack-nova-common-2014.1.2-1.el6.noarch

  At compute.log:

  2014-10-10 14:37:39.372 24897 ERROR nova.openstack.common.threadgroup [-] Unexpected vif_type=binding_failed
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     x.wait()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 173, in wait
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 121, in wait
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 293, in switch
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 212, in main
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 486, in run_service
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     service.start()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/service.py", line 163, in start
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     self.manager.init_host()
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1044, in init_host
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     self._init_instance(context, instance)
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 902, in _init_instance
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     self.driver.plug_vifs(instance, net_info)
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 860, in plug_vifs
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     self.vif_driver.plug(instance, vif)
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/vif.py", line 616, in plug
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup     _("Unexpected vif_type=%s") % vif_type)
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup NovaException: Unexpected vif_type=binding_failed
  2014-10-10 14:37:39.372 24897 TRACE nova.openstack.common.threadgroup

  
  At ovs-vswitchd.log:

  2014-10-10T06:37:38Z|00377|dpif|WARN|Dropped 9 log messages in last 17953 seconds (most recently, 17952 seconds ago) due to excessive rate
  2014-10-10T06:37:38Z|00378|dpif|WARN|system@ovs-system: port_del failed (Invalid argument)
  2014-10-10T06:37:38Z|00379|netdev_linux|WARN|Dropped 8 log messages in last 17953 seconds (most recently, 17952 seconds ago) due to excessive rate
  2014-10-10T06:37:38Z|00380|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:38Z|00381|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device
  2014-10-10T06:37:38Z|00382|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:38Z|00383|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device
  2014-10-10T06:37:38Z|00384|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:38Z|00385|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device
  2014-10-10T06:37:44Z|00386|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:44Z|00387|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device
  2014-10-10T06:37:44Z|00388|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:44Z|00389|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device
  2014-10-10T06:37:44Z|00390|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:44Z|00391|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device
  2014-10-10T06:37:44Z|00392|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:44Z|00393|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device
  2014-10-10T06:37:44Z|00394|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:44Z|00395|dpif|WARN|system@ovs-system: failed to add qvoeda2db74-94 as port: No such device
  2014-10-10T06:37:44Z|00396|bridge|INFO|bridge br-tun: using datapath ID 00003666e785084d
  2014-10-10T06:37:44Z|00397|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt"
  2014-10-10T06:37:44Z|00398|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:44Z|00399|dpif|WARN|system@ovs-system: failed to add qvoeae48b5f-9b as port: No such device
  2014-10-10T06:37:44Z|00400|bridge|INFO|bridge br-tun: added interface br-tun on port 65534
  2014-10-10T06:37:44Z|00401|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:44Z|00402|bridge|INFO|bridge br-int: added interface patch-tun on port 1
  2014-10-10T06:37:44Z|00403|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:44Z|00404|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:44Z|00405|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:44Z|00406|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:44Z|00407|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:44Z|00408|bridge|INFO|bridge br-tun: added interface patch-int on port 1
  2014-10-10T06:37:45Z|00409|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvo1a5fea4a-6e failed: No such device
  2014-10-10T06:37:45Z|00410|dpif|WARN|Dropped 6 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate
  2014-10-10T06:37:45Z|00411|dpif|WARN|system@ovs-system: failed to add qvo1a5fea4a-6e as port: No such device
  2014-10-10T06:37:45Z|00412|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeda2db74-94 failed: No such device
  2014-10-10T06:37:45Z|00413|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device qvoeae48b5f-9b failed: No such device
  2014-10-10T06:37:45Z|00414|bridge|INFO|bridge br-tun: added interface gre-c0a80547 on port 2
  2014-10-10T06:37:48Z|00415|ofproto|INFO|br-int: 3 flow_mods 10 s ago (2 adds, 1 deletes)
  2014-10-10T06:37:54Z|00416|ofproto|INFO|br-tun: 13 flow_mods in the 1 s starting 10 s ago (11 adds, 1 deletes, 1 modifications)

  But then these ports are already running under br-int -

  # ovs-vsctl show

  8fa6592e-b814-4c90-8ce5-8ee2f2e61d39
      Bridge br-int
          fail_mode: secure
          Port "qvoeda2db74-94"
              tag: 4095
              Interface "qvoeda2db74-94"
          Port patch-tun
              Interface patch-tun
                  type: patch
                  options: {peer=patch-int}
          Port "qvo2beb3ffb-6d"
              tag: 1
              Interface "qvo2beb3ffb-6d"
          Port "qvo1a5fea4a-6e"
              tag: 1
              Interface "qvo1a5fea4a-6e"
          Port "qvo684ea2ea-63"
              tag: 1
              Interface "qvo684ea2ea-63"
          Port "qvoeae48b5f-9b"
              tag: 1
              Interface "qvoeae48b5f-9b"
          Port br-int
              Interface br-int
                  type: internal
      Bridge br-tun
          Port patch-int
              Interface patch-int
                  type: patch
                  options: {peer=patch-tun}
          Port br-tun
              Interface br-tun
                  type: internal
          Port "gre-c0a80547"
              Interface "gre-c0a80547"
                  type: gre
                  options: {in_key=flow, local_ip="192.168.5.72", out_key=flow, remote_ip="192.168.5.71"}
      ovs_version: "1.11.0"

  Regards,
  Boon Lee

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1379663/+subscriptions
References

[Bug 1379663] [NEW] After upgrading - ovs-vswitchd cannot add existing ports
From: Boon Lee, 2014-10-10