yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1178568] Re: possible race condition during bare metal deploys

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Robert Collins <1178568@xxxxxxxxxxxxxxxxxx>
Date: Fri, 10 May 2013 11:05:29 -0000
Reply-to: Bug 1178568 <1178568@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
ok, this is a nonissue; the issue was the deploy-helper opaque error.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1178568

Title:
  possible race condition during bare metal deploys

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  There were no obvious failures during the deploy. This is what was
  logged:

  sudo grep 5f305000-ddf2-44b6-be74-059c88cf2109 /var/log/upstart/*
  /var/log/upstart/nova-api.log:2013-05-10 07:52:11,743.743 14328 INFO nova.osapi_compute.wsgi.server [req-ced5c8c5-f5b7-4158-9690-d9f59c590932 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] 10.10.16.134 "GET /v2/c30643483cd849a6b61e98c6ccc03a66/servers/5f305000-ddf2-44b6-be74-059c88cf2109 HTTP/1.1" status: 200 len: 1986 time: 0.0977318
  /var/log/upstart/nova-baremetal-deploy-helper.log:2013-05-10 07:51:31,210.210 14224 INFO nova.virt.baremetal.deploy_helper [-] request is queued: node 69, params {'swap_mb': 1, 'iqn': 'iqn-5f305000-ddf2-44b6-be74-059c88cf2109', 'image_path': u'/var/lib/nova/instances/instance-00000096/disk', 'address': '10.10.16.173', 'pxe_config_path': u'/tftpboot/5f305000-ddf2-44b6-be74-059c88cf2109/config', 'port': '3260', 'lun': '1', 'root_mb': 10240}
  /var/log/upstart/nova-baremetal-deploy-helper.log:2013-05-10 07:51:31,210.210 14224 INFO nova.virt.baremetal.deploy_helper [-] start deployment for node 69, params {'swap_mb': 1, 'iqn': 'iqn-5f305000-ddf2-44b6-be74-059c88cf2109', 'image_path': u'/var/lib/nova/instances/instance-00000096/disk', 'address': '10.10.16.173', 'pxe_config_path': u'/tftpboot/5f305000-ddf2-44b6-be74-059c88cf2109/config', 'port': '3260', 'lun': '1', 'root_mb': 10240}
  /var/log/upstart/nova-compute.log:2013-05-10 07:44:03,597.597 23016 AUDIT nova.compute.manager [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Starting instance...
  /var/log/upstart/nova-compute.log:2013-05-10 07:44:07,319.319 23016 AUDIT nova.compute.claims [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Attempting claim: memory 512 MB, disk 10 GB, VCPUs 1
  /var/log/upstart/nova-compute.log:2013-05-10 07:44:07,319.319 23016 AUDIT nova.compute.claims [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Total Memory: 98304 MB, used: 0 MB
  /var/log/upstart/nova-compute.log:2013-05-10 07:44:07,319.319 23016 AUDIT nova.compute.claims [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Memory limit: 98304 MB, free: 98304 MB
  /var/log/upstart/nova-compute.log:2013-05-10 07:44:07,320.320 23016 AUDIT nova.compute.claims [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Total Disk: 2048 GB, used: 0 GB
  /var/log/upstart/nova-compute.log:2013-05-10 07:44:07,320.320 23016 AUDIT nova.compute.claims [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Disk limit not specified, defaulting to unlimited
  /var/log/upstart/nova-compute.log:2013-05-10 07:44:07,321.321 23016 AUDIT nova.compute.claims [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Total CPU: 24 VCPUs, used: 0 VCPUs
  /var/log/upstart/nova-compute.log:2013-05-10 07:44:07,321.321 23016 AUDIT nova.compute.claims [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] CPU limit not specified, defaulting to unlimited
  /var/log/upstart/nova-compute.log:2013-05-10 07:44:07,322.322 23016 AUDIT nova.compute.claims [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Claim successful
  /var/log/upstart/nova-compute.log:2013-05-10 07:47:57,976.976 23016 INFO nova.compute.manager [-] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] During sync_power_state the instance has a pending task. Skip.
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:31,864.864 23016 ERROR nova.virt.baremetal.driver [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] Error deploying instance 5f305000-ddf2-44b6-be74-059c88cf2109 on baremetal node 1509a907-f0b6-471f-a0f5-7e2414cc896f.
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:36,528.528 23016 ERROR nova.compute.manager [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Instance failed to spawn
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:36,528.528 23016 TRACE nova.compute.manager [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Traceback (most recent call last):
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:36,528.528 23016 TRACE nova.compute.manager [instance: 5f305000-ddf2-44b6-be74-059c88cf2109]   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1200, in _spawn
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:36,528.528 23016 TRACE nova.compute.manager [instance: 5f305000-ddf2-44b6-be74-059c88cf2109]     block_device_info)
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:36,528.528 23016 TRACE nova.compute.manager [instance: 5f305000-ddf2-44b6-be74-059c88cf2109]   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 267, in spawn
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:36,528.528 23016 TRACE nova.compute.manager [instance: 5f305000-ddf2-44b6-be74-059c88cf2109]     self.power_off(instance, node)
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:36,528.528 23016 TRACE nova.compute.manager [instance: 5f305000-ddf2-44b6-be74-059c88cf2109]   File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 330, in power_off
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:36,528.528 23016 TRACE nova.compute.manager [instance: 5f305000-ddf2-44b6-be74-059c88cf2109]     "for instance %r") % instance['uuid'])
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:36,528.528 23016 TRACE nova.compute.manager [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] InstancePowerOffFailure: Baremetal power manager failed to stop node for instance u'5f305000-ddf2-44b6-be74-059c88cf2109'
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:36,528./var/log/upstart/nova-compute.log:2013-05-10 07:51:37,307.307 23016 AUDIT nova.compute.manager [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Terminating instance
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:50,840.840 23016 ERROR nova.virt.baremetal.driver [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] Error from baremetal driver during destroy: Baremetal power manager failed to stop node for instance u'5f305000-ddf2-44b6-be74-059c88cf2109'
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:50,922.922 23016 ERROR nova.compute.manager [req-e10b52ef-c85b-446b-9fa8-fa7394c081dc 87b44714e906420a9e6a07f6835b5b61 c30643483cd849a6b61e98c6ccc03a66] [instance: 5f305000-ddf2-44b6-be74-059c88cf2109] Error: ['Traceback (most recent call last):\n', '  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 942, in _build_instance\n    set_access_ip=set_access_ip)\n', '  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1204, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1200, in _spawn\n    block_device_info)\n', '  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 267, in spawn\n    self.power_off(instance, node)\n', '  File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 330, in power_off\n    "for instance %r") % instance[\'uuid\'])\n', "InstancePowerOffFailure: Baremetal power manager failed to stop node for instance u'5f305000-ddf2-44b6-be74-059c88cf2109'\n"]
  /var/log/upstart/nova-compute.log:2013-05-10 07:51:51,547.547 23016 TRACE nova.openstack.common.rpc.amqp InstancePowerOffFailure: Baremetal power manager failed to stop node for instance u'5f305000-ddf2-44b6-be74-059c88cf2109'528 23016 TRACE nova.compute.manager [instance: 5f305000-ddf2-44b6-be74-059c88cf2109]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1178568/+subscriptions