yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1675676] Re: Migrate VM failed when anti-affinity group was removed.

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Mon, 06 Nov 2017 05:26:59 -0000
Reply-to: Bug 1675676 <1675676@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
*** This bug is a duplicate of bug 1719730 ***
    https://bugs.launchpad.net/bugs/1719730

Looking at the code, the RequestSpec.from_components method is called
without an instance_group value (None is passed in):

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/conductor/tasks/migrate.py#L50

Which means the RequestSpec instance group is generated from the legacy
filter properties:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/objects/request_spec.py#L384

And the group_members value comes out of the filter properties dict
here:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/objects/request_spec.py#L188

And since the group was delete, there are no group members, so we get
the TypeError.

Before calling RequestSpec.from_components, conductor calls this method
to set instance group information in the request spec:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/conductor/tasks/migrate.py#L43

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/scheduler/utils.py#L335

However, we'd only set group_members in filter_properties if we were
able to lookup the group from the database:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/scheduler/utils.py#L353

And if you deleted the group, I'm not sure how we'd get there.

In other words, I'm not sure how this is True:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/objects/request_spec.py#L183

Unless the filter_properties dict has some old information in it?

Ah yes, it's due to this:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/objects/request_spec.py#L300

Called starting here:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/conductor/tasks/migrate.py#L42

That was fixed here for bug 1719730:

https://github.com/openstack/nova/commit/9d6632a67d91fb3c5145c14ac38011e919d6d8c0

So it looks like that needs to be backported, although newton is nearly
end of life at this point.

** This bug has been marked a duplicate of bug 1719730
   Reschedule after the late affinity check fails with "'NoneType' object is not iterable"

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1675676

Title:
  Migrate VM failed when anti-affinity group was removed.

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========
  Migrate VMs failed when anti-affinity group was removed from VMs.
  When anti-affinity group was removed, group_hosts of request_spec is set to an empty list (here's what I got when debugging nova-conductor: {'limits': {}, 'group_policies': set([u'anti-affinity']), 'ignore_hosts': [u'test-deploy7'], 'group_hosts': set([]), 'group_updated': True, 'scheduler_hints': {u'group': u'groupId'}}), then in nova.objects.request_spec.RequestSpec#_populate_group_info line 188: hosts = list(filter_properties.get('group_hosts')), and "TypeError: 'NoneType' object is not iterable" exception was raised.

  Steps to reproduce
  ==================
  1. Create an anti-affinity group named A;
  2. Create two VMs and assign these two VM into the anti-affinity group A created in previous step;
  3. Remove anti-affinity group A;
  4. Migrate these two VMs, and all of them got failed.

  Expected result
  ===============
  These two VMs were migrated successfully.

  Actual result
  =============
  Migration of these two VMs got failed.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
      OpenStack Newton.

  2. Which hypervisor did you use?
     Libvirt

  2. Which storage type did you use?
      Ceph

  3. Which networking type did you use?
     Neutron with OpenVSwitch

  Logs & Configs
  ==============
  2017-03-23 15:15:44.081 41242 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: c8f3a7be34c145a0a6921770994d5fae __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions [req-16b1e2ec-9a77-48dc-ade0-980650973089 142d8663efce464c89811c63e45bd82e f21a9c86d7114bf99c711f4874d80474 - 9f95b9967b894c928880feb32fad1d0d 9f95b9967b894c928880feb32fad1d0d] Unexpected exception in API method
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/migrate_server.py", line 52, in _migrate
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     self.compute_api.resize(req.environ['nova.context'], instance)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 166, in inner
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 173, in _wrapped
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     return fn(self, context, instance, *args, **kwargs)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 147, in inner
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3033, in resize
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     request_spec=request_spec)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 192, in resize_instance
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     request_spec=request_spec)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 317, in migrate_server
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     return cctxt.call(context, 'migrate_server', **kw)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     retry=self.retry)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     retry=retry)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     raise result
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions TypeError: 'NoneType' object is not iterable
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     res = self.dispatcher.dispatch(message)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     result = func(ctxt, **new_args)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 220, in migrate_server
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     reservations, clean_shutdown, request_spec)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 283, in _cold_migrate
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     updates, ex, legacy_spec)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     self.force_reraise()
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 252, in _cold_migrate
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     task.execute()
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in wrap
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     self.rollback()
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     self.force_reraise()
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in wrap
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     return original(self)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in execute
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     return self._execute()
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/migrate.py", line 54, in _execute
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     self.instance.availability_zone)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/request_spec.py", line 385, in from_components
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     spec_obj._populate_group_info(filter_properties)
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/request_spec.py", line 188, in _populate_group_info
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions     members = list(filter_properties.get('group_members'))
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions TypeError: 'NoneType' object is not iterable
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
  2017-03-23 15:15:44.084 41242 INFO nova.api.openstack.wsgi [req-16b1e2ec-9a77-48dc-ade0-980650973089 142d8663efce464c89811c63e45bd82e f21a9c86d7114bf99c711f4874d80474 - 9f95b9967b894c928880feb32fad1d0d 9f95b9967b894c928880feb32fad1d0d] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'>

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1675676/+subscriptions
References

[Bug 1675676] [NEW] Migrate VM failed when anti-affinity group was removed.
From: bryan, 2017-03-24