yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1202449] Re: Race condition with spawn/delete of instance sharing same source group security rule

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Thu, 04 Sep 2014 17:49:29 -0000
Reply-to: Bug 1202449 <1202449@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
*** This bug is a duplicate of bug 1182131 ***
    https://bugs.launchpad.net/bugs/1182131

I believe this was probably addressed with:

commit 6aa368b99249d01f8fd7183c15d11986ad6a6fb7
Author: Dan Smith <dansmith@xxxxxxxxxx>
Date:   Thu Jul 3 08:09:39 2014 -0700

    Avoid re-adding iptables rules for instances that have disappeared
    
    The remove_filters_for_instance() method fails silently if the
    instance's chain is gone (i.e. it's been deleted). If this
    happens while we're refreshing security group rules, we will
    not notice this case and re-add stale rules for an old instance,
    breaking our firewall for new instances.
    
    This adds a quick check after we've captured the lock to see if
    the associated chain exists, and bails if it doesn't.
    
    Change-Id: Ic75988939f82de49735d85fe99a9eecd4baf45c9
    Related-bug: #1182131


** Changed in: nova
       Status: Incomplete => Fix Committed

** This bug has been marked a duplicate of bug 1182131
   nova-compute: instance created in self-referencing secgroup produces KeyError

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1202449

Title:
  Race condition with spawn/delete of instance sharing same source group
  security rule

Status in OpenStack Compute (Nova):
  Fix Committed

Bug description:
  Getting the below error when launching and instance at the same time as deleting another instance. 
  These instances share a security group rule that has the same source group (hope that makes sense)

  
  Instance  e5fc8a20-384a-4976-890d-54631962c9e2 was deleted at about the same time as  98fd38d1-82cb-449e-92f9-1936d1e2628c was created.

  
   98fd38d1-82cb-449e-92f9-1936d1e2628c failed when trying to build its iptables because e5fc8a20-384a-4976-890d-54631962c9e2  no longer existed. 


  
  2013-07-17 15:30:35.754 ERROR nova.compute.manager [req-6f82cdb7-db20-4f17-a344-edf6da70a871 XXXX XXXXX] [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] Instance failed to spawn
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] Traceback (most recent call last):
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1119, in spawn
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] block_device_info)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1528, in spawn
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] block_device_info)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2443, in _create_domain_and_network
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] self.firewall_driver.prepare_instance_filter(instance, network_info)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 193, in prepare_instance_filter
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] ipv4_rules, ipv6_rules = self.instance_rules(instance, network_info)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 424, in instance_rules
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] conductor_api=capi)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 102, in wrapped
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] return func(self, context, *args, **kwargs)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 375, in get_instance_nw_info
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] result = self._get_instance_nw_info(context, instance)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 392, in _get_instance_nw_info
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] nw_info = self.network_rpcapi.get_instance_nw_info(context, **args)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 207, in get_instance_nw_info
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] project_id=project_id), version='1.9')
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 80, in call
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] return rpc.call(context, self._get_topic(topic), msg, timeout)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/_init__.py", line 140, in call
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] return _get_impl().call(CONF, context, topic, msg, timeout)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 800, in call
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] rpc_amqp.get_connection_pool(conf, Connection))
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 612, in call
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] rv = list(rv)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 561, in iter
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] raise result
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] InstanceNotFound_Remote: Instance e5fc8a20-384a-4976-890d-54631962c9e2 could not be found.
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] Traceback (most recent call last):
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] rval = self.proxy.dispatch(ctxt, version, method, **args)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] return getattr(proxyobj, method)(ctxt, **kwargs)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 576, in get_instance_nw_info
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] instance_uuid)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 549, in virtual_interface_get_by_instance
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] return IMPL.virtual_interface_get_by_instance(context, instance_id)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 114, in wrapper
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] return f(*args, **kwargs)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 126, in wrapper
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] instance_get_by_uuid(context, instance_uuid)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 114, in wrapper
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] return f(*args, **kwargs)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1607, in instance_get_by_uuid
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] return _instance_get_by_uuid(context, uuid)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 114, in wrapper
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] return f(*args, **kwargs)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1617, in _instance_get_by_uuid
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] raise exception.InstanceNotFound(instance_id=uuid)
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c] InstanceNotFound: Instance e5fc8a20-384a-4976-890d-54631962c9e2 could not be found.
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]
  2013-07-17 15:30:35.754 26853 TRACE nova.compute.manager [instance: 98fd38d1-82cb-449e-92f9-1936d1e2628c]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1202449/+subscriptions