yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1418298] Re: After service deleted, the corresponding compute-node can't restart again

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Fri, 20 Mar 2015 07:37:35 -0000
Reply-to: Bug 1418298 <1418298@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1418298

Title:
  After service deleted, the corresponding compute-node can't restart
  again

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  After remove a stopped service for nova-compute, then I can't restart
  it again. The error as below

  
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 145, in wait
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     x.wait()
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 47, in wait
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 173, in wait
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 293, in switch
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/service.py", line 492, in run_service
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     service.start()
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/service.py", line 181, in start
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/manager.py", line 1181, in pre_start_hook
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/manager.py", line 6058, in update_available_resource
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     rt.update_available_resource(context)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 342, in update_available_resource
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     return self._update_available_resource(context, resources)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 431, in inner
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 396, in _update_available_resource
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     self._sync_compute_node(context, resources)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 417, in _sync_compute_node
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     self._create(context, resources)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 466, in _create
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     values)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/conductor/api.py", line 170, in compute_node_create
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     return self._manager.compute_node_create(context, values)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 271, in compute_node_create
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     return cctxt.call(context, 'compute_node_create', values=values)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     retry=self.retry)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     timeout=timeout, retry=retry)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     retry=retry)
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 399, in _send
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup     raise result

  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup RemoteError: Remote error: DBDuplicateEntry (IntegrityError) (1062, "Duplicate entry 'hp-pc-hp-pc' for key 'uniq_compute_nodes0host0hypervisor_hostname'") 'INSERT INTO compute_nodes (created_at, updated_at, deleted_at, deleted, service_id, host, vcpus, memory_mb, local_gb, vcpus_used, memory_mb_used, local_gb_used, hypervisor_type, hypervisor_version, hypervisor_hostname, free_ram_mb, free_disk_gb, current_workload, running_vms, cpu_info, disk_available_least, host_ip, supported_instances, metrics, pci_stats, extra_resources, stats, numa_topology) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' (datetime.datetime(2015, 2, 5, 3, 9, 3, 245004), None, None, 0, 6, 'hp-pc', 8, 7920, 45, 0, 512, 0, 'QEMU', 2000000, 'hp-pc', 7408, 45, 0, 0, '{"vendor": "Intel", "model": "Nehalem", "arch": "x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme", "dtes64", "tsc", "vmx", "xtpr", "cmov", "ssse3", "est", "pat", "monitor", "smx", "lm", "msr", "nx", "fxsr", "tm", "sse4.1", "pae", "sse4.2", "acpi", "de", "mmx", "cx8", "mce", "mtrr", "rdtscp", "ht", "pse", "lahf_lm", "pdcm", "mca", "apic", "sse", "ds", "pni", "tm2", "sse2", "ss", "pbe", "fpu", "cx16", "pse36", "ds_cpl", "popcnt"], "topology": {"cores": 4, "threads": 2, "sockets": 1}}', 34, '10.238.154.76', '[["alpha", "qemu", "hvm"], ["armv7l", "qemu", "hvm"], ["cris", "qemu", "hvm"], ["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["lm32", "qemu", "hvm"], ["m68k", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["mips64", "qemu", "hvm"], ["mips64el", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["ppcemb", "qemu", "hvm"], ["s390x", "qemu", "hvm"], ["sh4", "qemu", "hvm"], ["sh4eb", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["sparc64", "qemu", "hvm"], ["unicore32", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"], ["xtensa", "qemu", "hvm"], ["xtensaeb", "qemu", "hvm"]]', '[]', '[]', None, '{}', '{"nova_object.version": "1.2", "nova_object.changes": ["cells"], "nova_object.name": "NUMATopology", "nova_object.data": {"cells": [{"nova_object.version": "1.2", "nova_object.changes": ["cpu_usage", "memory_usage", "cpuset", "pinned_cpus", "siblings", "memory", "mempages", "id"], "nova_object.name": "NUMACell", "nova_object.data": {"cpu_usage": 0, "memory_usage": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7], "pinned_cpus": [], "siblings": [[0, 1], [6, 7], [2, 4], [3, 5]], "memory": 7920, "mempages": [], "id": 0}, "nova_object.namespace": "nova"}]}, "nova_object.namespace": "nova"}')
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup [u'Traceback (most recent call last):\n', u'  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', u'  File "/opt/stack/nova/nova/conductor/manager.py", line 313, in compute_node_create\n    result = self.db.compute_node_create(context, values)\n', u'  File "/opt/stack/nova/nova/db/api.py", line 247, in compute_node_create\n    return IMPL.compute_node_create(context, values)\n', u'  File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 128, in wrapper\n    return f(*args, **kwargs)\n', u'  File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 599, in compute_node_create\n    compute_node_ref.save()\n', u'  File "/opt/stack/nova/nova/db/sqlalchemy/models.py", line 82, in save\n    super(NovaBase, self).save(session=session)\n', u'  File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save\n    session.flush()\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1919, in flush\n    self._flush(objects)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2037, in _flush\n    transaction.rollback(_capture_exception=True)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2001, in _flush\n    flush_context.execute()\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute\n    rec.execute(self)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute\n    uow\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 65, in save_obj\n    mapper, table, insert)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 602, in _emit_insert_statements\n    execute(statement, params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute\n    return meth(self, multiparams, params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection\n    return connection._execute_clauseelement(self, multiparams, params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement\n    compiled_sql, distilled_params\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context\n    context)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception\n    util.raise_from_cause(newraise, exc_info)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context\n    context)\n', u'  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute\n    cursor.execute(statement, parameters)\n', u'  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute\n    self.errorhandler(self, exc, value)\n', u'  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler\n    raise errorclass, errorvalue\n', u'DBDuplicateEntry: (IntegrityError) (1062, "Duplicate entry \'hp-pc-hp-pc\' for key \'uniq_compute_nodes0host0hypervisor_hostname\'") \'INSERT INTO compute_nodes (created_at, updated_at, deleted_at, deleted, service_id, host, vcpus, memory_mb, local_gb, vcpus_used, memory_mb_used, local_gb_used, hypervisor_type, hypervisor_version, hypervisor_hostname, free_ram_mb, free_disk_gb, current_workload, running_vms, cpu_info, disk_available_least, host_ip, supported_instances, metrics, pci_stats, extra_resources, stats, numa_topology) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)\' (datetime.datetime(2015, 2, 5, 3, 9, 3, 245004), None, None, 0, 6, \'hp-pc\', 8, 7920, 45, 0, 512, 0, \'QEMU\', 2000000, \'hp-pc\', 7408, 45, 0, 0, \'{"vendor": "Intel", "model": "Nehalem", "arch": "x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme", "dtes64", "tsc", "vmx", "xtpr", "cmov", "ssse3", "est", "pat", "monitor", "smx", "lm", "msr", "nx", "fxsr", "tm", "sse4.1", "pae", "sse4.2", "acpi", "de", "mmx", "cx8", "mce", "mtrr", "rdtscp", "ht", "pse", "lahf_lm", "pdcm", "mca", "apic", "sse", "ds", "pni", "tm2", "sse2", "ss", "pbe", "fpu", "cx16", "pse36", "ds_cpl", "popcnt"], "topology": {"cores": 4, "threads": 2, "sockets": 1}}\', 34, \'10.238.154.76\', \'[["alpha", "qemu", "hvm"], ["armv7l", "qemu", "hvm"], ["cris", "qemu", "hvm"], ["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["lm32", "qemu", "hvm"], ["m68k", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["mips64", "qemu", "hvm"], ["mips64el", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["ppcemb", "qemu", "hvm"], ["s390x", "qemu", "hvm"], ["sh4", "qemu", "hvm"], ["sh4eb", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["sparc64", "qemu", "hvm"], ["unicore32", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"], ["xtensa", "qemu", "hvm"], ["xtensaeb", "qemu", "hvm"]]\', \'[]\', \'[]\', None, \'{}\', \'{"nova_object.version": "1.2", "nova_object.changes": ["cells"], "nova_object.name": "NUMATopology", "nova_object.data": {"cells": [{"nova_object.version": "1.2", "nova_object.changes": ["cpu_usage", "memory_usage", "cpuset", "pinned_cpus", "siblings", "memory", "mempages", "id"], "nova_object.name": "NUMACell", "nova_object.data": {"cpu_usage": 0, "memory_usage": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7], "pinned_cpus": [], "siblings": [[0, 1], [6, 7], [2, 4], [3, 5]], "memory": 7920, "mempages": [], "id": 0}, "nova_object.namespace": "nova"}]}, "nova_object.namespace": "nova"}\')\n'].
  2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1418298/+subscriptions
References

[Bug 1418298] [NEW] After service deleted, then corresponding compute-node can't restart again
From: Alex Xu, 2015-02-05