yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1685150] [NEW] Nova evacuate silently ignoring rebuild error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sanjay Kumar Singh <ersanjay.singh@xxxxxxxxx>
Date: Fri, 21 Apr 2017 08:49:20 -0000
Reply-to: Bug 1685150 <1685150@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Instance stuck in rebuild state during evacuate process. below is the
log of compute node, looks like nova compute silently ignore the error.

2017-04-21 13:50:57.155 2537 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: d9075b4dea404be387a57e3c6ef18543 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:299
2017-04-21 13:50:57.159 2537 DEBUG oslo_messaging._drivers.amqpdriver [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] CALL msg_id: 404b30ae4ac34f37aedce3e8f357c04d exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:442
2017-04-21 13:50:57.177 2537 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 404b30ae4ac34f37aedce3e8f357c04d __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:299
2017-04-21 13:50:57.178 2537 DEBUG nova.scheduler.client.report [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] Refreshing aggregate associations for resource provider 0da9f49c-8861-41a6-bf5d-1f6553413408 _ensure_resource_provider /usr/lib/python2.7/site-packages/nova/scheduler/client/report.py:402
2017-04-21 13:50:57.199 2537 DEBUG oslo_concurrency.lockutils [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker.drop_move_claim" :: held 0.052s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-04-21 13:50:57.202 2537 DEBUG oslo_messaging._drivers.amqpdriver [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] CALL msg_id: 9b86d0d1439b443581a1273dd9ec486e exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:442
2017-04-21 13:50:57.218 2537 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 9b86d0d1439b443581a1273dd9ec486e __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:299
2017-04-21 13:50:57.219 2537 DEBUG nova.compute.manager [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] [instance: afab8b57-f5ea-468c-9948-cfe359026b19] Instance was deleted while rebuilding rebuild_instance /usr/lib/python2.7/site-packages/nova/compute/manager.py:2767
2017-04-21 13:50:57.220 2537 DEBUG oslo_messaging._drivers.amqpdriver [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] CALL msg_id: d9a233f63bd7499c90c97461b725bccf exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:442
2017-04-21 13:50:57.233 2537 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: d9a233f63bd7499c90c97461b725bccf __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:299
2017-04-21 13:50:57.234 2537 DEBUG nova.compute.utils [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] [instance: afab8b57-f5ea-468c-9948-cfe359026b19] Instance afab8b57-f5ea-468c-9948-cfe359026b19 could not be found. notify_about_instance_usage /usr/lib/python2.7/site-packages/nova/compute/utils.py:316
2017-04-21 13:50:57.245 2537 DEBUG oslo_messaging._drivers.amqpdriver [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] CAST unique_id: 9b396335358e4df28aba76628cd57157 NOTIFY exchange 'nova' topic 'notifications.error' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:426

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1685150

Title:
  Nova evacuate silently ignoring rebuild error

Status in OpenStack Compute (nova):
  New

Bug description:
  Instance stuck in rebuild state during evacuate process. below is the
  log of compute node, looks like nova compute silently ignore the
  error.

  2017-04-21 13:50:57.155 2537 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: d9075b4dea404be387a57e3c6ef18543 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:299
  2017-04-21 13:50:57.159 2537 DEBUG oslo_messaging._drivers.amqpdriver [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] CALL msg_id: 404b30ae4ac34f37aedce3e8f357c04d exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:442
  2017-04-21 13:50:57.177 2537 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 404b30ae4ac34f37aedce3e8f357c04d __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:299
  2017-04-21 13:50:57.178 2537 DEBUG nova.scheduler.client.report [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] Refreshing aggregate associations for resource provider 0da9f49c-8861-41a6-bf5d-1f6553413408 _ensure_resource_provider /usr/lib/python2.7/site-packages/nova/scheduler/client/report.py:402
  2017-04-21 13:50:57.199 2537 DEBUG oslo_concurrency.lockutils [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker.drop_move_claim" :: held 0.052s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
  2017-04-21 13:50:57.202 2537 DEBUG oslo_messaging._drivers.amqpdriver [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] CALL msg_id: 9b86d0d1439b443581a1273dd9ec486e exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:442
  2017-04-21 13:50:57.218 2537 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 9b86d0d1439b443581a1273dd9ec486e __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:299
  2017-04-21 13:50:57.219 2537 DEBUG nova.compute.manager [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] [instance: afab8b57-f5ea-468c-9948-cfe359026b19] Instance was deleted while rebuilding rebuild_instance /usr/lib/python2.7/site-packages/nova/compute/manager.py:2767
  2017-04-21 13:50:57.220 2537 DEBUG oslo_messaging._drivers.amqpdriver [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] CALL msg_id: d9a233f63bd7499c90c97461b725bccf exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:442
  2017-04-21 13:50:57.233 2537 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: d9a233f63bd7499c90c97461b725bccf __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:299
  2017-04-21 13:50:57.234 2537 DEBUG nova.compute.utils [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] [instance: afab8b57-f5ea-468c-9948-cfe359026b19] Instance afab8b57-f5ea-468c-9948-cfe359026b19 could not be found. notify_about_instance_usage /usr/lib/python2.7/site-packages/nova/compute/utils.py:316
  2017-04-21 13:50:57.245 2537 DEBUG oslo_messaging._drivers.amqpdriver [req-a6580f6b-cc80-4c41-9e8a-939e8136aa30 - - - - -] CAST unique_id: 9b396335358e4df28aba76628cd57157 NOTIFY exchange 'nova' topic 'notifications.error' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:426

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1685150/+subscriptions
Follow ups

[Bug 1685150] Re: Nova evacuate silently ignoring rebuild error
From: Launchpad Bug Tracker, 2017-06-24