yahoo-eng-team team mailing list archive

[Vish Ishaya <vishvananda@xxxxxxxxx>]

*** This bug is a duplicate of bug 1334142 ***
    https://bugs.launchpad.net/bugs/1334142

** This bug is no longer a duplicate of bug 1368344
   Failure in test_server_actions
** This bug has been marked a duplicate of bug 1334142
   A server creation fails due to adding interface failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1334272

Title:
  Server fails to become active because: Failed to add interface: Dump
  was interrupted and may be inconsistent

Status in OpenStack Compute (Nova):
  New

Bug description:
  Since the trusty upgrade of nodes we're seeing a new timing issue
  where computes don't build due to a race around network interface
  adding. The n-cpu traceback that we see in these cases is:

  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
      readers.get(fileno, noop).cb(fileno)
    File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
      result = function(*args, **kwargs)
    File "/opt/stack/new/nova/nova/compute/manager.py", line 1544, in _allocate_network_async
      dhcp_options=dhcp_options)
    File "/opt/stack/new/nova/nova/network/api.py", line 46, in wrapped
      return func(self, context, *args, **kwargs)
    File "/opt/stack/new/nova/nova/network/base_api.py", line 61, in wrapper
      res = f(self, context, *args, **kwargs)
    File "/opt/stack/new/nova/nova/network/api.py", line 253, in allocate_for_instance
      nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
    File "/opt/stack/new/nova/nova/network/rpcapi.py", line 175, in allocate_for_instance
      macs=jsonutils.to_primitive(macs))
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
      retry=self.retry)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
      timeout=timeout, retry=retry)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 401, in send
      retry=retry)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 392, in _send
      raise result
  NovaException_Remote: Failed to add interface: Dump was interrupted and may be inconsistent.

  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
      incoming.message))

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
      result = getattr(endpoint, method)(ctxt, **new_args)

    File "/opt/stack/new/nova/nova/network/floating_ips.py", line 118, in allocate_for_instance
      **kwargs)

    File "/opt/stack/new/nova/nova/network/manager.py", line 525, in allocate_for_instance
      requested_networks=requested_networks)

    File "/opt/stack/new/nova/nova/network/manager.py", line 213, in _allocate_fixed_ips
      vpn=vpn, address=address)

    File "/opt/stack/new/nova/nova/network/manager.py", line 945, in allocate_fixed_ip
      exc_info=True)

    File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/opt/stack/new/nova/nova/network/manager.py", line 927, in allocate_fixed_ip
      self._setup_network_on_host(context, network)

    File "/opt/stack/new/nova/nova/network/manager.py", line 1746, in _setup_network_on_host
      self.l3driver.initialize_gateway(network)

    File "/opt/stack/new/nova/nova/network/l3.py", line 104, in initialize_gateway
      gateway=(network_ref['gateway'] is not None))

    File "/opt/stack/new/nova/nova/network/linux_net.py", line 1380, in plug
      return _get_interface_driver().plug(network, mac_address, gateway)

    File "/opt/stack/new/nova/nova/network/linux_net.py", line 1428, in plug
      network, gateway)

    File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 249, in inner
      return f(*args, **kwargs)

    File "/opt/stack/new/nova/nova/network/linux_net.py", line 1571, in ensure_bridge
      raise exception.NovaException(msg)

  NovaException: Failed to add interface: Dump was interrupted and may
  be inconsistent.

  
  Removing descriptor: 3

  
  An example failure is here - http://logs.openstack.org/23/95723/8/check/check-tempest-dsvm-postgres-full/e1494ef/logs/screen-n-cpu.txt.gz?level=TRACE

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1334272/+subscriptions