yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1279642] Re: Fail to delete an instance with the TaskState 'deleting'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Wed, 01 Apr 2015 13:39:47 -0000
Reply-to: Bug 1279642 <1279642@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I believe this is a failed db upgrade

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1279642

Title:
  Fail to delete an instance with the TaskState 'deleting'

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  When I try to delete an instance with the Task State ' deleting', I
  got the following error:

   ERROR nova.api.openstack [req-05ccd477-2931-4cac-9921-86c904e87e65 demo demo] Caught error: (OperationalError) (1054, "Unknown column 'compute_nodes_1.stats' in 'field list'") 'SELECT anon_1.services_created_at AS anon_1_services_created_at, anon_1.services_updated_at AS anon_1_services_updated_at, anon_1.services_deleted_at AS anon_1_services_deleted_at, anon_1.services_deleted AS anon_1_services_deleted, anon_1.services_id AS anon_1_services_id, anon_1.services_host AS anon_1_services_host, anon_1.services_binary AS anon_1_services_binary, anon_1.services_topic AS anon_1_services_topic, anon_1.services_report_count AS anon_1_services_report_count, anon_1.services_disabled AS anon_1_services_disabled, anon_1.services_disabled_reason AS anon_1_services_disabled_reason, compute_nodes_1.created_at AS compute_nodes_1_created_at, compute_nodes_1.updated_at AS compute_nodes_1_updated_at, compute_nodes_1.deleted_at AS compute_nodes_1_deleted_at, compute_nodes_1.deleted AS compute_nodes_1_deleted, compute_nodes_1.id AS compute_nodes_1_id, compute_nodes_1.service_id AS compute_nodes_1_service_id, compute_nodes_1.vcpus AS compute_nodes_1_vcpus, compute_nodes_1.memory_mb AS compute_nodes_1_memory_mb, compute_nodes_1.local_gb AS compute_nodes_1_local_gb, compute_nodes_1.vcpus_used AS compute_nodes_1_vcpus_used, compute_nodes_1.memory_mb_used AS compute_nodes_1_memory_mb_used, compute_nodes_1.local_gb_used AS compute_nodes_1_local_gb_used, compute_nodes_1.hypervisor_type AS compute_nodes_1_hypervisor_type, compute_nodes_1.hypervisor_version AS compute_nodes_1_hypervisor_version, compute_nodes_1.hypervisor_hostname AS compute_nodes_1_hypervisor_hostname, compute_nodes_1.free_ram_mb AS compute_nodes_1_free_ram_mb, compute_nodes_1.free_disk_gb AS compute_nodes_1_free_disk_gb, compute_nodes_1.current_workload AS compute_nodes_1_current_workload, compute_nodes_1.running_vms AS compute_nodes_1_running_vms, compute_nodes_1.cpu_info AS compute_nodes_1_cpu_info, compute_nodes_1.disk_available_least AS compute_nodes_1_disk_available_least, compute_nodes_1.host_ip AS compute_nodes_1_host_ip, compute_nodes_1.supported_instances AS compute_nodes_1_supported_instances, compute_nodes_1.metrics AS compute_nodes_1_metrics, compute_nodes_1.pci_stats AS compute_nodes_1_pci_stats, compute_nodes_1.extra_resources AS compute_nodes_1_extra_resources, compute_nodes_1.stats AS compute_nodes_1_stats \nFROM (SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason \nFROM services \nWHERE services.deleted = %s AND services.host = %s AND services.topic = %s \n LIMIT %s) AS anon_1 LEFT OUTER JOIN compute_nodes AS compute_nodes_1 ON compute_nodes_1.service_id = anon_1.services_id AND compute_nodes_1.deleted = %s' (0, 'xu-dev', 'compute', 1, 0)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack Traceback (most recent call last):
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return req.get_response(self.application)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     application, catch_exc_info=False)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 600, in __call__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return self.app(env, start_response)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 929, in __call__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     content_type, body, accept)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 991, in _process_stack
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1078, in dispatch
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return method(req=request, **action_args)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 1184, in delete
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     self._delete(req.environ['nova.context'], req, id)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 1015, in _delete
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     self.compute_api.delete(context, instance)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 198, in wrapped
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return func(self, context, target, *args, **kwargs)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 188, in inner
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return function(self, context, instance, *args, **kwargs)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 215, in _wrapped
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return fn(self, context, instance, *args, **kwargs)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 169, in inner
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return f(self, context, instance, *args, **kw)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 1670, in delete
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     self._delete_instance(context, instance)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 1660, in _delete_instance
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     task_state=task_states.DELETING)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 1493, in _delete
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     user_id=user_id)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     six.reraise(self.type_, self.value, self.tb)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 1450, in _delete
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     context.elevated(), instance.host)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/base.py", line 112, in wrapper
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     result = fn(cls, context, *args, **kwargs)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/service.py", line 100, in get_by_compute_host
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     db_service = db.service_get_by_compute_host(context, host)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/api.py", line 124, in service_get_by_compute_host
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return IMPL.service_get_by_compute_host(context, host)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 110, in wrapper
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return f(*args, **kwargs)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 450, in service_get_by_compute_host
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     filter_by(topic=CONF.compute_topic).\
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2156, in first
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     ret = list(self[0:1])
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2023, in __getitem__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return list(res)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     return self._execute_and_instances(context)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     result = conn.execute(querycontext.statement, self._params)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     params)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     compiled_sql, distilled_params
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     context)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     context)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     cursor.execute(statement, parameters)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     self.errorhandler(self, exc, value)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2014-02-13 22:20:18.152 TRACE nova.api.openstack     raise errorclass, errorvalue
  2014-02-13 22:20:18.152 TRACE nova.api.openstack OperationalError: (OperationalError) (1054, "Unknown column 'compute_nodes_1.stats' in 'field list'") 'SELECT anon_1.services_created_at AS anon_1_services_created_at, anon_1.services_updated_at AS anon_1_services_updated_at, anon_1.services_deleted_at AS anon_1_services_deleted_at, anon_1.services_deleted AS anon_1_services_deleted, anon_1.services_id AS anon_1_services_id, anon_1.services_host AS anon_1_services_host, anon_1.services_binary AS anon_1_services_binary, anon_1.services_topic AS anon_1_services_topic, anon_1.services_report_count AS anon_1_services_report_count, anon_1.services_disabled AS anon_1_services_disabled, anon_1.services_disabled_reason AS anon_1_services_disabled_reason, compute_nodes_1.created_at AS compute_nodes_1_created_at, compute_nodes_1.updated_at AS compute_nodes_1_updated_at, compute_nodes_1.deleted_at AS compute_nodes_1_deleted_at, compute_nodes_1.deleted AS compute_nodes_1_deleted, compute_nodes_1.id AS compute_nodes_1_id, compute_nodes_1.service_id AS compute_nodes_1_service_id, compute_nodes_1.vcpus AS compute_nodes_1_vcpus, compute_nodes_1.memory_mb AS compute_nodes_1_memory_mb, compute_nodes_1.local_gb AS compute_nodes_1_local_gb, compute_nodes_1.vcpus_used AS compute_nodes_1_vcpus_used, compute_nodes_1.memory_mb_used AS compute_nodes_1_memory_mb_used, compute_nodes_1.local_gb_used AS compute_nodes_1_local_gb_used, compute_nodes_1.hypervisor_type AS compute_nodes_1_hypervisor_type, compute_nodes_1.hypervisor_version AS compute_nodes_1_hypervisor_version, compute_nodes_1.hypervisor_hostname AS compute_nodes_1_hypervisor_hostname, compute_nodes_1.free_ram_mb AS compute_nodes_1_free_ram_mb, compute_nodes_1.free_disk_gb AS compute_nodes_1_free_disk_gb, compute_nodes_1.current_workload AS compute_nodes_1_current_workload, compute_nodes_1.running_vms AS compute_nodes_1_running_vms, compute_nodes_1.cpu_info AS compute_nodes_1_cpu_info, compute_nodes_1.disk_available_least AS compute_nodes_1_disk_available_least, compute_nodes_1.host_ip AS compute_nodes_1_host_ip, compute_nodes_1.supported_instances AS compute_nodes_1_supported_instances, compute_nodes_1.metrics AS compute_nodes_1_metrics, compute_nodes_1.pci_stats AS compute_nodes_1_pci_stats, compute_nodes_1.extra_resources AS compute_nodes_1_extra_resources, compute_nodes_1.stats AS compute_nodes_1_stats \nFROM (SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason \nFROM services \nWHERE services.deleted = %s AND services.host = %s AND services.topic = %s \n LIMIT %s) AS anon_1 LEFT OUTER JOIN compute_nodes AS compute_nodes_1 ON compute_nodes_1.service_id = anon_1.services_id AND compute_nodes_1.deleted = %s' (0, '-dev', 'compute', 1, 0)
  2014-02-13 22:20:18.152 TRACE nova.api.openstack

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1279642/+subscriptions
References

[Bug 1279642] [NEW] Fail to delete an instance with the TastState 'deleting'
From: Haiwei Xu, 2014-02-13