yahoo-eng-team team mailing list archive

Thread
Date

[Bug 1334272] [NEW] Server fails to become active because: Failed to add interface: Dump was interrupted and may be inconsistent

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Wed, 25 Jun 2014 13:06:18 -0000
Reply-to: Bug 1334272 <1334272@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

*** This bug is a duplicate of bug 1334142 ***
    https://bugs.launchpad.net/bugs/1334142

Public bug reported:

Since the trusty upgrade of nodes we're seeing a new timing issue where
computes don't build due to a race around network interface adding. The
n-cpu traceback that we see in these cases is:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/new/nova/nova/compute/manager.py", line 1544, in _allocate_network_async
    dhcp_options=dhcp_options)
  File "/opt/stack/new/nova/nova/network/api.py", line 46, in wrapped
    return func(self, context, *args, **kwargs)
  File "/opt/stack/new/nova/nova/network/base_api.py", line 61, in wrapper
    res = f(self, context, *args, **kwargs)
  File "/opt/stack/new/nova/nova/network/api.py", line 253, in allocate_for_instance
    nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
  File "/opt/stack/new/nova/nova/network/rpcapi.py", line 175, in allocate_for_instance
    macs=jsonutils.to_primitive(macs))
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
    retry=self.retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
    timeout=timeout, retry=retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 401, in send
    retry=retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 392, in _send
    raise result
NovaException_Remote: Failed to add interface: Dump was interrupted and may be inconsistent.

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
    incoming.message))

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/opt/stack/new/nova/nova/network/floating_ips.py", line 118, in allocate_for_instance
    **kwargs)

  File "/opt/stack/new/nova/nova/network/manager.py", line 525, in allocate_for_instance
    requested_networks=requested_networks)

  File "/opt/stack/new/nova/nova/network/manager.py", line 213, in _allocate_fixed_ips
    vpn=vpn, address=address)

  File "/opt/stack/new/nova/nova/network/manager.py", line 945, in allocate_fixed_ip
    exc_info=True)

  File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/opt/stack/new/nova/nova/network/manager.py", line 927, in allocate_fixed_ip
    self._setup_network_on_host(context, network)

  File "/opt/stack/new/nova/nova/network/manager.py", line 1746, in _setup_network_on_host
    self.l3driver.initialize_gateway(network)

  File "/opt/stack/new/nova/nova/network/l3.py", line 104, in initialize_gateway
    gateway=(network_ref['gateway'] is not None))

  File "/opt/stack/new/nova/nova/network/linux_net.py", line 1380, in plug
    return _get_interface_driver().plug(network, mac_address, gateway)

  File "/opt/stack/new/nova/nova/network/linux_net.py", line 1428, in plug
    network, gateway)

  File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 249, in inner
    return f(*args, **kwargs)

  File "/opt/stack/new/nova/nova/network/linux_net.py", line 1571, in ensure_bridge
    raise exception.NovaException(msg)

NovaException: Failed to add interface: Dump was interrupted and may be
inconsistent.


Removing descriptor: 3


An example failure is here - http://logs.openstack.org/23/95723/8/check/check-tempest-dsvm-postgres-full/e1494ef/logs/screen-n-cpu.txt.gz?level=TRACE

** Affects: nova
     Importance: Undecided
         Status: New

** This bug has been marked a duplicate of bug 1334142
   A server creation fails due to adding interface failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1334272

Title:
  Server fails to become active because: Failed to add interface: Dump
  was interrupted and may be inconsistent

Status in OpenStack Compute (Nova):
  New

Bug description:
  Since the trusty upgrade of nodes we're seeing a new timing issue
  where computes don't build due to a race around network interface
  adding. The n-cpu traceback that we see in these cases is:

  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
      readers.get(fileno, noop).cb(fileno)
    File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
      result = function(*args, **kwargs)
    File "/opt/stack/new/nova/nova/compute/manager.py", line 1544, in _allocate_network_async
      dhcp_options=dhcp_options)
    File "/opt/stack/new/nova/nova/network/api.py", line 46, in wrapped
      return func(self, context, *args, **kwargs)
    File "/opt/stack/new/nova/nova/network/base_api.py", line 61, in wrapper
      res = f(self, context, *args, **kwargs)
    File "/opt/stack/new/nova/nova/network/api.py", line 253, in allocate_for_instance
      nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
    File "/opt/stack/new/nova/nova/network/rpcapi.py", line 175, in allocate_for_instance
      macs=jsonutils.to_primitive(macs))
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
      retry=self.retry)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
      timeout=timeout, retry=retry)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 401, in send
      retry=retry)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 392, in _send
      raise result
  NovaException_Remote: Failed to add interface: Dump was interrupted and may be inconsistent.

  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
      incoming.message))

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
      result = getattr(endpoint, method)(ctxt, **new_args)

    File "/opt/stack/new/nova/nova/network/floating_ips.py", line 118, in allocate_for_instance
      **kwargs)

    File "/opt/stack/new/nova/nova/network/manager.py", line 525, in allocate_for_instance
      requested_networks=requested_networks)

    File "/opt/stack/new/nova/nova/network/manager.py", line 213, in _allocate_fixed_ips
      vpn=vpn, address=address)

    File "/opt/stack/new/nova/nova/network/manager.py", line 945, in allocate_fixed_ip
      exc_info=True)

    File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/opt/stack/new/nova/nova/network/manager.py", line 927, in allocate_fixed_ip
      self._setup_network_on_host(context, network)

    File "/opt/stack/new/nova/nova/network/manager.py", line 1746, in _setup_network_on_host
      self.l3driver.initialize_gateway(network)

    File "/opt/stack/new/nova/nova/network/l3.py", line 104, in initialize_gateway
      gateway=(network_ref['gateway'] is not None))

    File "/opt/stack/new/nova/nova/network/linux_net.py", line 1380, in plug
      return _get_interface_driver().plug(network, mac_address, gateway)

    File "/opt/stack/new/nova/nova/network/linux_net.py", line 1428, in plug
      network, gateway)

    File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 249, in inner
      return f(*args, **kwargs)

    File "/opt/stack/new/nova/nova/network/linux_net.py", line 1571, in ensure_bridge
      raise exception.NovaException(msg)

  NovaException: Failed to add interface: Dump was interrupted and may
  be inconsistent.

  
  Removing descriptor: 3

  
  An example failure is here - http://logs.openstack.org/23/95723/8/check/check-tempest-dsvm-postgres-full/e1494ef/logs/screen-n-cpu.txt.gz?level=TRACE

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1334272/+subscriptions

Follow ups

[Bug 1334272] Re: Server fails to become active because: Failed to add interface: Dump was interrupted and may be inconsistent
From: Vish Ishaya, 2014-09-11
[Bug 1334272] Re: Server fails to become active because: Failed to add interface: Dump was interrupted and may be inconsistent
From: Vish Ishaya, 2014-09-11
[Bug 1334272] [NEW] Server fails to become active because: Failed to add interface: Dump was interrupted and may be inconsistent
From: Sean Dague, 2014-06-25

References

[Bug 1334272] [NEW] Server fails to become active because: Failed to add interface: Dump was interrupted and may be inconsistent
From: Sean Dague, 2014-06-25