yahoo-eng-team team mailing list archive

[Christopher Yeoh <cyeoh@xxxxxxxxxxx>]

Public bug reported:

Sometimes servers fail to startup - intermittent gate/check bug

Example error:

2014-01-28 02:14:36.309 ERROR nova.scheduler.filter_scheduler [req-a617cb8a-4f34-42f9-a4a7-e951cf538219 InstanceActionsTestJSON-tempest-857832860-user InstanceActionsTestJSON-tempest-857832860-tenant] [instance: 0f9850c1-b5a8-4d62-98ab-11b04ba41165] Error from last host: devstack-precise-1390529664 (node devstack-precise-1390529664): [u'Traceback (most recent call last):
', u'  File "/opt/stack/new/nova/nova/compute/manager.py", line 1068, in _build_instance
    set_access_ip=set_access_ip)
', u'  File "/opt/stack/new/nova/nova/compute/manager.py", line 354, in decorated_function
    return function(self, context, *args, **kwargs)
', u'  File "/opt/stack/new/nova/nova/compute/manager.py", line 1478, in _spawn
    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)
', u'  File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
', u'  File "/opt/stack/new/nova/nova/compute/manager.py", line 1475, in _spawn
    block_device_info)
', u'  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2187, in spawn
    admin_pass=admin_password)
', u'  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2490, in _create_image
    project_id=instance[\'project_id\'])
', u'  File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 180, in cache
    imagecache.refresh_timestamp(base)
', u'  File "/opt/stack/new/nova/nova/virt/libvirt/imagecache.py", line 261, in refresh_timestamp
    inner_refresh_timestamp()
', u'  File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 249, in inner
    return f(*args, **kwargs)
', u'  File "/opt/stack/new/nova/nova/virt/libvirt/imagecache.py", line 258, in inner_refresh_timestamp
    os.utime(base_file, None)
', u"OSError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/_base/307971e863bbda9ebaf13fcfedcb13498a5dbde6'
"]

>From here:

http://logs.openstack.org/91/58191/4/check/check-tempest-dsvm-
full/0dc8d4d/logs/screen-n-sch.txt.gz#_2014-01-28_02_14_36_309

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1273556

Title:
  Servers sometimes fail to startup in the gate with a permission denied

Status in OpenStack Compute (Nova):
  New

Bug description:
  Sometimes servers fail to startup - intermittent gate/check bug

  Example error:

  2014-01-28 02:14:36.309 ERROR nova.scheduler.filter_scheduler [req-a617cb8a-4f34-42f9-a4a7-e951cf538219 InstanceActionsTestJSON-tempest-857832860-user InstanceActionsTestJSON-tempest-857832860-tenant] [instance: 0f9850c1-b5a8-4d62-98ab-11b04ba41165] Error from last host: devstack-precise-1390529664 (node devstack-precise-1390529664): [u'Traceback (most recent call last):
  ', u'  File "/opt/stack/new/nova/nova/compute/manager.py", line 1068, in _build_instance
      set_access_ip=set_access_ip)
  ', u'  File "/opt/stack/new/nova/nova/compute/manager.py", line 354, in decorated_function
      return function(self, context, *args, **kwargs)
  ', u'  File "/opt/stack/new/nova/nova/compute/manager.py", line 1478, in _spawn
      LOG.exception(_(\'Instance failed to spawn\'), instance=instance)
  ', u'  File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
      six.reraise(self.type_, self.value, self.tb)
  ', u'  File "/opt/stack/new/nova/nova/compute/manager.py", line 1475, in _spawn
      block_device_info)
  ', u'  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2187, in spawn
      admin_pass=admin_password)
  ', u'  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2490, in _create_image
      project_id=instance[\'project_id\'])
  ', u'  File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 180, in cache
      imagecache.refresh_timestamp(base)
  ', u'  File "/opt/stack/new/nova/nova/virt/libvirt/imagecache.py", line 261, in refresh_timestamp
      inner_refresh_timestamp()
  ', u'  File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 249, in inner
      return f(*args, **kwargs)
  ', u'  File "/opt/stack/new/nova/nova/virt/libvirt/imagecache.py", line 258, in inner_refresh_timestamp
      os.utime(base_file, None)
  ', u"OSError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/_base/307971e863bbda9ebaf13fcfedcb13498a5dbde6'
  "]

  From here:

  http://logs.openstack.org/91/58191/4/check/check-tempest-dsvm-
  full/0dc8d4d/logs/screen-n-sch.txt.gz#_2014-01-28_02_14_36_309

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1273556/+subscriptions