yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1478430] [NEW] nova failed to list old db instances

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Eli Qiao <liyong.qiao@xxxxxxxxx>
Date: Mon, 27 Jul 2015 04:01:36 -0000
Reply-to: Bug 1478430 <1478430@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I have a old version of nova, there is an instance running, when I upgrade to
lastest upstream version of nova (commit 183cd88cb2f9781b53f71b6b161df401c286c9ff)

after sync db, I get 500 error when I try to list the old instance.

nova-api get follow error:

2015-07-27 12:00:11.397 TRACE nova.api.openstack     sort_keys=sort_keys, sort_dirs=sort_dirs)
2015-07-27 12:00:11.397 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 2083, in _get_instances_by_filters
2015-07-27 12:00:11.397 TRACE nova.api.openstack     expected_attrs=fields, sort_keys=sort_keys, sort_dirs=sort_dirs)
2015-07-27 12:00:11.397 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/base.py", line 71, in wrapper
2015-07-27 12:00:11.397 TRACE nova.api.openstack     result = fn(cls, context, *args, **kwargs)
2015-07-27 12:00:11.397 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 1221, in get_by_filters
2015-07-27 12:00:11.397 TRACE nova.api.openstack     expected_attrs)
2015-07-27 12:00:11.397 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 1146, in _make_instance_list
2015-07-27 12:00:11.397 TRACE nova.api.openstack     expected_attrs=expected_attrs)
2015-07-27 12:00:11.397 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 505, in _from_db_object
2015-07-27 12:00:11.397 TRACE nova.api.openstack     if db_inst['info_cache'] is None:
2015-07-27 12:00:11.397 TRACE nova.api.openstack KeyError: 'info_cache'
2015-07-27 12:00:11.397 TRACE nova.api.openstack 


nova compute get follow error when restart:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/queue.py", line 117, in switch
    self.greenlet.switch(value)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 623, in run_service
    service.start()
  File "/opt/stack/nova/nova/service.py", line 158, in start
    self.manager.init_host()
  File "/opt/stack/nova/nova/compute/manager.py", line 1261, in init_host
    context, self.host, expected_attrs=['info_cache'])
  File "/opt/stack/nova/nova/objects/base.py", line 69, in wrapper
    args, kwargs)
  File "/opt/stack/nova/nova/conductor/rpcapi.py", line 243, in object_class_action
    objver=objver, args=args, kwargs=kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
    retry=self.retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
    timeout=timeout, retry=retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
    retry=retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
    raise result
KeyError: u'\'info_cache\'\nTraceback (most recent call last):\n\n  File "/opt/stack/nova/nova/conductor/manager.py", line 440, in _object_dispatch\n    return getattr(target, method)(*args, **kwargs)\n\n  File "/opt/stack/nova/nova/objects/base.py", line 71, in wrapper\n    result = fn(cls, context, *args, **kwargs)\n\n  File "/opt/stack/nova/nova/objects/instance.py", line 1229, in get_by_host\n    expected_attrs)\n\n  File "/opt/stack/nova/nova/objects/instance.py", line 1146, in _make_instance_list\n    expected_attrs=expected_attrs)\n\n  File "/opt/stack/nova/nova/objects/instance.py", line 505, in _from_db_object\n    if db_inst[\'info_cache\'] is None:\n\nKeyError: \'info_cache\'\n'


for reference, the old db_inst is follow:

{'vm_state': u'active', 'internal_id': None, 'availability_zone': None,
'terminated_at': None, 'ramdisk_id': u'd73b7b35-b89b-4e92-98fc-
dc5a7929f214', 'instance_type_id': 2L, 'updated_at':
datetime.datetime(2015, 7, 27, 3, 49, 29), 'cleaned': 0L, 'vm_mode':
None, 'deleted_at': None, 'reservation_id': u'r-0s8p28s9', 'id': 76L,
'disable_terminate': False, 'user_id':
u'bea64c5634ae4f079c1c69ed29a68242', 'uuid': u'25976bc8-cd6f-471f-
a7d4-7ce7614f9b9e', 'default_swap_device': None, 'hostname': u'test2',
'launched_on': u'liyong', 'display_description': u'test2', 'key_data':
None, 'deleted': 0L, 'power_state': 1L, 'default_ephemeral_device':
None, 'progress': 0L, 'project_id': u'52a0ee83fb524376bd603547aea415a3',
'launched_at': datetime.datetime(2015, 7, 27, 3, 49, 29),
'scheduled_at': None, 'node': u'liyong', 'ephemeral_gb': 0L,
'access_ip_v6': None, 'access_ip_v4': None, 'kernel_id': u'712e417f-
16ec-42fb-b2cb-ef8c316d9903', 'key_name': None, 'user_data': None,
'host': u'liyong', 'root_gb': 1L, 'architecture': None, 'display_name':
u'test2', 'system_metadata':
[<nova.db.sqlalchemy.models.InstanceSystemMetadata object at
0x7ff3aef724d0>, <nova.db.sqlalchemy.models.InstanceSystemMetadata
object at 0x7ff3aef7f1d0>,
<nova.db.sqlalchemy.models.InstanceSystemMetadata object at
0x7ff3aef7f310>, <nova.db.sqlalchemy.models.InstanceSystemMetadata
object at 0x7ff3aef7f3d0>,
<nova.db.sqlalchemy.models.InstanceSystemMetadata object at
0x7ff3aef7f490>, <nova.db.sqlalchemy.models.InstanceSystemMetadata
object at 0x7ff3aef7f550>,
<nova.db.sqlalchemy.models.InstanceSystemMetadata object at
0x7ff3aef7f610>], 'task_state': None, 'shutdown_terminate': False,
'cell_name': None, 'ephemeral_key_uuid': None, 'locked': False, 'name':
'instance-0000004c', 'created_at': datetime.datetime(2015, 7, 27, 3, 49,
18), 'locked_by': None, 'launch_index': 0L, 'metadata': [], 'memory_mb':
512L, 'vcpus': 1L, 'image_ref': u'93169d18-1bb2-45f7-861c-6735ef687db9',
'root_device_name': u'/dev/vda', 'auto_disk_config': False, 'os_type':
None, 'config_drive': u''}

** Affects: nova
     Importance: Undecided
     Assignee: Eli Qiao (taget-9)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Eli Qiao (taget-9)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1478430

Title:
  nova failed to list old db instances

Status in OpenStack Compute (nova):
  New

Bug description:
  I have a old version of nova, there is an instance running, when I upgrade to
  lastest upstream version of nova (commit 183cd88cb2f9781b53f71b6b161df401c286c9ff)

  after sync db, I get 500 error when I try to list the old instance.

  nova-api get follow error:

  2015-07-27 12:00:11.397 TRACE nova.api.openstack     sort_keys=sort_keys, sort_dirs=sort_dirs)
  2015-07-27 12:00:11.397 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 2083, in _get_instances_by_filters
  2015-07-27 12:00:11.397 TRACE nova.api.openstack     expected_attrs=fields, sort_keys=sort_keys, sort_dirs=sort_dirs)
  2015-07-27 12:00:11.397 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/base.py", line 71, in wrapper
  2015-07-27 12:00:11.397 TRACE nova.api.openstack     result = fn(cls, context, *args, **kwargs)
  2015-07-27 12:00:11.397 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 1221, in get_by_filters
  2015-07-27 12:00:11.397 TRACE nova.api.openstack     expected_attrs)
  2015-07-27 12:00:11.397 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 1146, in _make_instance_list
  2015-07-27 12:00:11.397 TRACE nova.api.openstack     expected_attrs=expected_attrs)
  2015-07-27 12:00:11.397 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 505, in _from_db_object
  2015-07-27 12:00:11.397 TRACE nova.api.openstack     if db_inst['info_cache'] is None:
  2015-07-27 12:00:11.397 TRACE nova.api.openstack KeyError: 'info_cache'
  2015-07-27 12:00:11.397 TRACE nova.api.openstack 

  
  nova compute get follow error when restart:

  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/eventlet/queue.py", line 117, in switch
      self.greenlet.switch(value)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
      result = function(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 623, in run_service
      service.start()
    File "/opt/stack/nova/nova/service.py", line 158, in start
      self.manager.init_host()
    File "/opt/stack/nova/nova/compute/manager.py", line 1261, in init_host
      context, self.host, expected_attrs=['info_cache'])
    File "/opt/stack/nova/nova/objects/base.py", line 69, in wrapper
      args, kwargs)
    File "/opt/stack/nova/nova/conductor/rpcapi.py", line 243, in object_class_action
      objver=objver, args=args, kwargs=kwargs)
    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
      retry=self.retry)
    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
      timeout=timeout, retry=retry)
    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
      retry=retry)
    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
      raise result
  KeyError: u'\'info_cache\'\nTraceback (most recent call last):\n\n  File "/opt/stack/nova/nova/conductor/manager.py", line 440, in _object_dispatch\n    return getattr(target, method)(*args, **kwargs)\n\n  File "/opt/stack/nova/nova/objects/base.py", line 71, in wrapper\n    result = fn(cls, context, *args, **kwargs)\n\n  File "/opt/stack/nova/nova/objects/instance.py", line 1229, in get_by_host\n    expected_attrs)\n\n  File "/opt/stack/nova/nova/objects/instance.py", line 1146, in _make_instance_list\n    expected_attrs=expected_attrs)\n\n  File "/opt/stack/nova/nova/objects/instance.py", line 505, in _from_db_object\n    if db_inst[\'info_cache\'] is None:\n\nKeyError: \'info_cache\'\n'


  for reference, the old db_inst is follow:

  {'vm_state': u'active', 'internal_id': None, 'availability_zone':
  None, 'terminated_at': None, 'ramdisk_id': u'd73b7b35-b89b-4e92-98fc-
  dc5a7929f214', 'instance_type_id': 2L, 'updated_at':
  datetime.datetime(2015, 7, 27, 3, 49, 29), 'cleaned': 0L, 'vm_mode':
  None, 'deleted_at': None, 'reservation_id': u'r-0s8p28s9', 'id': 76L,
  'disable_terminate': False, 'user_id':
  u'bea64c5634ae4f079c1c69ed29a68242', 'uuid': u'25976bc8-cd6f-471f-
  a7d4-7ce7614f9b9e', 'default_swap_device': None, 'hostname': u'test2',
  'launched_on': u'liyong', 'display_description': u'test2', 'key_data':
  None, 'deleted': 0L, 'power_state': 1L, 'default_ephemeral_device':
  None, 'progress': 0L, 'project_id':
  u'52a0ee83fb524376bd603547aea415a3', 'launched_at':
  datetime.datetime(2015, 7, 27, 3, 49, 29), 'scheduled_at': None,
  'node': u'liyong', 'ephemeral_gb': 0L, 'access_ip_v6': None,
  'access_ip_v4': None, 'kernel_id': u'712e417f-16ec-42fb-b2cb-
  ef8c316d9903', 'key_name': None, 'user_data': None, 'host': u'liyong',
  'root_gb': 1L, 'architecture': None, 'display_name': u'test2',
  'system_metadata': [<nova.db.sqlalchemy.models.InstanceSystemMetadata
  object at 0x7ff3aef724d0>,
  <nova.db.sqlalchemy.models.InstanceSystemMetadata object at
  0x7ff3aef7f1d0>, <nova.db.sqlalchemy.models.InstanceSystemMetadata
  object at 0x7ff3aef7f310>,
  <nova.db.sqlalchemy.models.InstanceSystemMetadata object at
  0x7ff3aef7f3d0>, <nova.db.sqlalchemy.models.InstanceSystemMetadata
  object at 0x7ff3aef7f490>,
  <nova.db.sqlalchemy.models.InstanceSystemMetadata object at
  0x7ff3aef7f550>, <nova.db.sqlalchemy.models.InstanceSystemMetadata
  object at 0x7ff3aef7f610>], 'task_state': None, 'shutdown_terminate':
  False, 'cell_name': None, 'ephemeral_key_uuid': None, 'locked': False,
  'name': 'instance-0000004c', 'created_at': datetime.datetime(2015, 7,
  27, 3, 49, 18), 'locked_by': None, 'launch_index': 0L, 'metadata': [],
  'memory_mb': 512L, 'vcpus': 1L, 'image_ref': u'93169d18-1bb2-45f7
  -861c-6735ef687db9', 'root_device_name': u'/dev/vda',
  'auto_disk_config': False, 'os_type': None, 'config_drive': u''}

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1478430/+subscriptions
Follow ups

[Bug 1478430] Re: nova failed to list old db instances
From: Thierry Carrez, 2015-09-24