yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1474550] Re: network allocation randomly failing with InstanceUpdateConflict after compare and swap was merged

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Tue, 28 Jul 2015 09:07:58 -0000
Reply-to: Bug 1474550 <1474550@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => liberty-2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1474550

Title:
  network allocation randomly failing with InstanceUpdateConflict after
  compare and swap was merged

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Seeing this quite a bit:

  http://logs.openstack.org/85/197185/10/gate/gate-tempest-dsvm-
  cells/7ef1949/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-07-14_16_39_06_862

  2015-07-14 16:39:06.862 ERROR nova.compute.manager [req-94a9749e-2aed-433c-8dea-f672b61b7fcf tempest-DeleteServersTestJSON-193542712 tempest-DeleteServersTestJSON-409686918] [instance: f143b952-9edd-4971-86fe-227d61294a76] Failed to allocate network(s)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] Traceback (most recent call last):
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2057, in _build_resources
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     requested_networks, security_groups)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1572, in _build_networks_for_instance
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     requested_networks, macs, security_groups, dhcp_options)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1606, in _allocate_network
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     instance.save(expected_task_state=[None])
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/objects/base.py", line 100, in wrapper
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     self._context, self, fn.__name__, args, kwargs)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 266, in object_action
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     objmethod=objmethod, args=args, kwargs=kwargs)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     retry=self.retry)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     timeout=timeout, retry=retry)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 361, in send
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     retry=retry)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 352, in _send
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     raise result
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] InstanceUpdateConflict_Remote: Conflict updating instance f143b952-9edd-4971-86fe-227d61294a76
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] Traceback (most recent call last):
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/conductor/manager.py", line 437, in _object_dispatch
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     return getattr(target, method)(*args, **kwargs)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/objects/base.py", line 116, in wrapper
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     return fn(self, *args, **kwargs)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/objects/instance.py", line 862, in save
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     columns_to_join=_expected_cols(expected_attrs))
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/db/api.py", line 751, in instance_update_and_get_original
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     columns_to_join=columns_to_join)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 231, in wrapper
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     return f(*args, **kwargs)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 146, in wrapper
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     ectxt.value = e.inner_exc
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     six.reraise(self.type_, self.value, self.tb)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in wrapper
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     return f(*args, **kwargs)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2452, in instance_update_and_get_original
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     original=instance_ref))
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2581, in _instance_update
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76]     raise exception.InstanceUpdateConflict(instance_uuid=instance_uuid)
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] InstanceUpdateConflict: Conflict updating instance f143b952-9edd-4971-86fe-227d61294a76
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 
  2015-07-14 16:39:06.862 3472 ERROR nova.compute.manager [instance: f143b952-9edd-4971-86fe-227d61294a76] 

  Ever since this was merged yesterday:

  https://review.openstack.org/#/c/141115/

  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSW5zdGFuY2VVcGRhdGVDb25mbGljdF9SZW1vdGVcIiBBTkQgdGFnczpcInNjcmVlbi1uLWNwdS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzNjkwNTIwMTM4NH0=

  101 hits in 24 hours, check and gate, all failures.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1474550/+subscriptions
References

[Bug 1474550] [NEW] network allocation randomly failing with InstanceUpdateConflict after compare and swap was merged
From: Matt Riedemann, 2015-07-14