yahoo-eng-team team mailing list archive

[Julia Kreger <juliaashleykreger@xxxxxxxxx>]

Public bug reported:

On a juno-2 TripleO based system, after performing a node upgrade via
`nova rebuild`, we encountered an issue where some virtual machines were
unreachable.  Upon further investigation, we found that the OVS VIF was
missing, and the root cause was that the re-creation failed as seen in
the log below.  The flow was never re-asserted as the logic in
nova/virt/libvirt/vif.py function plug_ovs_hybrid only executes the
steps that would result in the VIF being recreated if the interface did
not already exist.  As the interface already existed upon attempting to
restart, it appears to have never been re-asserted hence requiring the
command to be manually executed to restore network connectivity for the
virtual machine.

# cat nova-compute.log
Traceback (most recent call last):
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 455, in fire_timers
    timer()
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
    result = function(*args, **kwargs)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/service.py", line 490, in run_service
    service.start()
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/service.py", line 164, in start
    self.manager.init_host()
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1059, in init_host
    self._init_instance(context, instance)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 918, in _init_instance
    self.driver.plug_vifs(instance, net_info)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 820, in plug_vifs
    self.vif_driver.plug(instance, vif)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 627, in plug
    self.plug_ovs(instance, vif)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 492, in plug_ovs
    self.plug_ovs_hybrid(instance, vif)
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 488, in plug_ovs_hybrid
    instance['uuid'])
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/linux_net.py", line 1320, in create_ovs_vif_port
    'external-ids:vm-uuid=%s' % instance_id])
  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/linux_net.py", line 1310, in _ovs_vsctl
    raise exception.AgentError(method=full_args)
AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', '--if-exists', 'del-port', u'qvo8670d393-d4', '--', 'add-port', u'br-int', u'qvo8670d393-d4', '--', 'set', 'Interface', u'qvo8670d393-d4', u'external-ids:iface-id=8670d393-d47e-41e1-b2b5-b94819266934', 'external-ids:iface-status=active', u'external-ids:attached-mac=fa:16:3e:b1:23:d1', 'external-ids:vm-uuid=26f03afc-e198-41d0-b7c4-77616f9bdfbe']

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1391699

Title:
  nova vm networking failed to initialize as step after interface
  creation failed.

Status in OpenStack Compute (Nova):
  New

Bug description:
  On a juno-2 TripleO based system, after performing a node upgrade via
  `nova rebuild`, we encountered an issue where some virtual machines
  were unreachable.  Upon further investigation, we found that the OVS
  VIF was missing, and the root cause was that the re-creation failed as
  seen in the log below.  The flow was never re-asserted as the logic in
  nova/virt/libvirt/vif.py function plug_ovs_hybrid only executes the
  steps that would result in the VIF being recreated if the interface
  did not already exist.  As the interface already existed upon
  attempting to restart, it appears to have never been re-asserted hence
  requiring the command to be manually executed to restore network
  connectivity for the virtual machine.

  # cat nova-compute.log
  Traceback (most recent call last):
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 455, in fire_timers
      timer()
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
      cb(*args, **kw)
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
      result = function(*args, **kwargs)
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/service.py", line 490, in run_service
      service.start()
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/service.py", line 164, in start
      self.manager.init_host()
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1059, in init_host
      self._init_instance(context, instance)
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 918, in _init_instance
      self.driver.plug_vifs(instance, net_info)
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 820, in plug_vifs
      self.vif_driver.plug(instance, vif)
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 627, in plug
      self.plug_ovs(instance, vif)
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 492, in plug_ovs
      self.plug_ovs_hybrid(instance, vif)
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 488, in plug_ovs_hybrid
      instance['uuid'])
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/linux_net.py", line 1320, in create_ovs_vif_port
      'external-ids:vm-uuid=%s' % instance_id])
    File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/linux_net.py", line 1310, in _ovs_vsctl
      raise exception.AgentError(method=full_args)
  AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', '--if-exists', 'del-port', u'qvo8670d393-d4', '--', 'add-port', u'br-int', u'qvo8670d393-d4', '--', 'set', 'Interface', u'qvo8670d393-d4', u'external-ids:iface-id=8670d393-d47e-41e1-b2b5-b94819266934', 'external-ids:iface-status=active', u'external-ids:attached-mac=fa:16:3e:b1:23:d1', 'external-ids:vm-uuid=26f03afc-e198-41d0-b7c4-77616f9bdfbe']

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1391699/+subscriptions