yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1279572] Re: DBError in nova-compute with baremetal driver after stats table change

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Wed, 05 Mar 2014 13:06:51 -0000
Reply-to: Bug 1279572 <1279572@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1279572

Title:
  DBError in nova-compute with baremetal driver after stats table change

Status in OpenStack Compute (Nova):
  Fix Released
Status in tripleo - openstack on openstack:
  Fix Released

Bug description:
  As of the following commit:
  https://github.com/openstack/nova/commit/8a7b95dccdbe449d5235868781b30edebd34bacd
  our nova-compute service on the seed node is throwing DBErrors.  If I
  reset my Nova tree to the previous commit and downgrade the database
  to 232 then I am able to use nova successfully again.  With that
  commit all boots fail with a No hosts found message, presumably
  related to the following messages in the nova-compute log:

  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/openstack/common/service.py", line 480, in run_service
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     service.start()
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/service.py", line 193, in start
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/compute/manager.py", line 835, in pre_start_hook
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/compute/manager.py", line 5121, in update_available_resource
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     rt.update_available_resource(context)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 353, in update_available_resource
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     self._sync_compute_node(context, resources)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 384, in _sync_compute_node
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     self._update(context, resources)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 456, in _update
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     context, self.compute_node, values)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/conductor/api.py", line 241, in compute_node_update
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     return self._manager.compute_node_update(context, node, values)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 364, in compute_node_update
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     prune_stats=prune_stats)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     wait_for_reply=True, timeout=timeout)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/oslo/messaging/transport.py", line 87, in _send
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     timeout=timeout)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 390, in send
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     return self._send(target, ctxt, message, wait_for_reply, timeout)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup   File "/opt/stack/venvs/nova/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 383, in _send
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup     raise result
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup RemoteError: Remote error: DBError (ProgrammingError) (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near \': "\'amd64\'", u\'baremetal_driver\': "\'nova.virt.baremetal.pxe.PXE\'"} WHERE compute\' at line 1') 'UPDATE compute_nodes SET updated_at=%s, vcpus_used=%s, memory_mb_used=%s, local_gb_used=%s, free_ram_mb=%s, free_disk_gb=%s, current_workload=%s, running_vms=%s, stats=%s WHERE compute_nodes.id = %s' (datetime.datetime(2014, 2, 12, 23, 8, 23, 932601), 0, 0, 0, 4096, 20, 0, 0, {u'cpu_arch': u'amd64', u'baremetal_driver': u'nova.virt.baremetal.pxe.PXE'}, 1L)
  Feb 12 23:08:24 localhost nova-compute: 2014-02-12 23:08:23.980 4572 TRACE nova.openstack.common.threadgroup [u'Traceback (most recent call last):\n', u'  File "/opt/stack/venvs/nova/lib/python2.7/site-packages/oslo/messaging/_executors/base.py", line 36, in _dispatch\n    incoming.reply(self.callback(incoming.ctxt, incoming.message))\n', u'  File "/opt/stack/venvs/nova/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in __call__\n    return self._dispatch(endpoint, method, ctxt, args)\n', u'  File "/opt/stack/venvs/nova/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', u'  File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/conductor/manager.py", line 458, in compute_node_update\n    result = self.db.compute_node_update(context, node[\'id\'], values)\n', u'  File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/db/api.py", line 228, in compute_node_update\n    return IMPL.compute_node_update(context, compute_id, values)\n', u'  File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 110, in wrapper\n    return f(*args, **kwargs)\n', u'  File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 166, in wrapped\n    return f(*args, **kwargs)\n', u'  File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 614, in compute_node_update\n    compute_ref.update(values)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 447, in __exit__\n    self.rollback()\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 444, in __exit__\n    self.commit()\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 354, in commit\n    self._prepare_impl()\n', u'  File "/usr/lib64/python2.7/site-packages/sqlalche
  Feb 12 23:08:24 localhost nova-compute: my/orm/session.py", line 334, in _prepare_impl\n    self.session.flush()\n', u'  File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 616, in _wrap\n    raise exception.DBError(e)\n', u'DBError: (ProgrammingError) (1064, \'You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near \\\': "\\\'amd64\\\'", u\\\'baremetal_driver\\\': "\\\'nova.virt.baremetal.pxe.PXE\\\'"} WHERE compute\\\' at line 1\') \'UPDATE compute_nodes SET updated_at=%s, vcpus_used=%s, memory_mb_used=%s, local_gb_used=%s, free_ram_mb=%s, free_disk_gb=%s, current_workload=%s, running_vms=%s, stats=%s WHERE compute_nodes.id = %s\' (datetime.datetime(2014, 2, 12, 23, 8, 23, 932601), 0, 0, 0, 4096, 20, 0, 0, {u\'cpu_arch\': u\'amd64\', u\'baremetal_driver\': u\'nova.virt.baremetal.pxe.PXE\'}, 1L)\n'].

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1279572/+subscriptions
References

[Bug 1279572] [NEW] DBError in nova-compute with baremetal driver after stats table change
From: Ben Nemec, 2014-02-12