yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1280034] [NEW] compute_node_update broken with havana compute nodes

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dan Smith <dms@xxxxxxxxxxxxx>
Date: Fri, 14 Feb 2014 00:34:09 -0000
Reply-to: Bug 1280034 <1280034@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

This change:

https://review.openstack.org/#/c/66469

Changed the format of the data in the "values" dictionary of
compute_node_update. This causes an icehouse conductor to generate a
broken SQL query when called from a havana compute node:

http://logs.openstack.org/75/64075/13/check/check-grenade-
dsvm/b70c839/logs/new/screen-n-cond.txt.gz?level=TRACE

executors.base [-] Exception during message handling: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ': '2', u'io_workload': '0', u'num_instances': '2', u'num_vm_building': '0', u'nu' at line 1") 'UPDATE compute_nodes SET updated_at=%s, vcpus_used=%s, memory_mb_used=%s, free_ram_mb=%s, running_vms=%s, stats=%s WHERE compute_nodes.id = %s' (datetime.datetime(2014, 2, 12, 21, 2, 12, 395978), 4, 1216, 6737, 4, {u'num_task_None': 2, u'io_workload': 0, u'num_instances': 2, u'num_vm_active': 1, u'num_task_scheduling': 0, u'num_vm_building': 0, u'num_proj_d0e1e781676f4fe5b1b81e31b8ae87de': 1, u'num_vcpus_used': 2, u'num_proj_a8a2f9c3e3bd44edb1c5fd2ae4cc7b3c': 1, u'num_os_type_None': 2, u'num_vm_error': 1}, 1)
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base Traceback (most recent call last):
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/oslo.messaging/oslo/messaging/_executors/base.py", line 36, in _dispatch
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     incoming.reply(self.callback(incoming.ctxt, incoming.message))
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 134, in __call__
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     return self._dispatch(endpoint, method, ctxt, args)
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     result = getattr(endpoint, method)(ctxt, **new_args)
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/conductor/manager.py", line 458, in compute_node_update
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     result = self.db.compute_node_update(context, node['id'], values)
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/db/api.py", line 228, in compute_node_update
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     return IMPL.compute_node_update(context, compute_id, values)
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 110, in wrapper
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     return f(*args, **kwargs)
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 166, in wrapped
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     return f(*args, **kwargs)
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 614, in compute_node_update
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     compute_ref.update(values)
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 456, in __exit__
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     self.commit()
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 368, in commit
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     self._prepare_impl()
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 347, in _prepare_impl
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     self.session.flush()
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/openstack/common/db/sqlalchemy/session.py", line 616, in _wrap
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     raise exception.DBError(e)
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base DBError: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ': '2', u'io_workload': '0', u'num_instances': '2', u'num_vm_building': '0', u'nu' at line 1") 'UPDATE compute_nodes SET updated_at=%s, vcpus_used=%s, memory_mb_used=%s, free_ram_mb=%s, running_vms=%s, stats=%s WHERE compute_nodes.id = %s' (datetime.datetime(2014, 2, 12, 21, 2, 12, 395978), 4, 1216, 6737, 4, {u'num_task_None': 2, u'io_workload': 0, u'num_instances': 2, u'num_vm_active': 1, u'num_task_scheduling': 0, u'num_vm_building': 0, u'num_proj_d0e1e781676f4fe5b1b81e31b8ae87de': 1, u'num_vcpus_used': 2, u'num_proj_a8a2f9c3e3bd44edb1c5fd2ae4cc7b3c': 1, u'num_os_type_None': 2, u'num_vm_error': 1}, 1)
2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base 
2014-02-12 21:02:12.404 18598 ERROR oslo.messaging._drivers.common [-] Returning exception (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ': '2', u'io_workload': '0', u'num_instances': '2', u'num_vm_building': '0', u'nu' at line 1") 'UPDATE compute_nodes SET updated_at=%s, vcpus_used=%s, memory_mb_used=%s, free_ram_mb=%s, running_vms=%s, stats=%s WHERE compute_nodes.id = %s' (datetime.datetime(2014, 2, 12, 21, 2, 12, 395978), 4, 1216, 6737, 4, {u'num_task_None': 2, u'io_workload': 0, u'num_instances': 2, u'num_vm_active': 1, u'num_task_scheduling': 0, u'num_vm_building': 0, u'num_proj_d0e1e781676f4fe5b1b81e31b8ae87de': 1, u'num_vcpus_used': 2, u'num_proj_a8a2f9c3e3bd44edb1c5fd2ae4cc7b3c': 1, u'num_os_type_None': 2, u'num_vm_error': 1}, 1) to caller
2014-02-12 21:02:12.404 18598 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/_executors/base.py", line 36, in _dispatch\n    incoming.reply(self.callback(incoming.ctxt, incoming.message))\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 134, in __call__\n    return self._dispatch(endpoint, method, ctxt, args)\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/opt/stack/new/nova/nova/conductor/manager.py", line 458, in compute_node_update\n    result = self.db.compute_node_update(context, node[\'id\'], values)\n', '  File "/opt/stack/new/nova/nova/db/api.py", line 228, in compute_node_update\n    return IMPL.compute_node_update(context, compute_id, values)\n', '  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 110, in wrapper\n    return f(*args, **kwargs)\n', '  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 166, in wrapped\n    return f(*args, **kwargs)\n', '  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 614, in compute_node_update\n    compute_ref.update(values)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 456, in __exit__\n    self.commit()\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 368, in commit\n    self._prepare_impl()\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 347, in _prepare_impl\n    self.session.flush()\n', '  File "/opt/stack/new/nova/nova/openstack/common/db/sqlalchemy/session.py", line 616, in _wrap\n    raise exception.DBError(e)\n', 'DBError: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \': \'2\', u\'io_workload\': \'0\', u\'num_instances\': \'2\', u\'num_vm_building\': \'0\', u\'nu\' at line 1") \'UPDATE compute_nodes SET updated_at=%s, vcpus_used=%s, memory_mb_used=%s, free_ram_mb=%s, running_vms=%s, stats=%s WHERE compute_nodes.id = %s\' (datetime.datetime(2014, 2, 12, 21, 2, 12, 395978), 4, 1216, 6737, 4, {u\'num_task_None\': 2, u\'io_workload\': 0, u\'num_instances\': 2, u\'num_vm_active\': 1, u\'num_task_scheduling\': 0, u\'num_vm_building\': 0, u\'num_proj_d0e1e781676f4fe5b1b81e31b8ae87de\': 1, u\'num_vcpus_used\': 2, u\'num_proj_a8a2f9c3e3bd44edb1c5fd2ae4cc7b3c\': 1, u\'num_os_type_None\': 2, u\'num_vm_error\': 1}, 1)\n']
2014-02-12 21:02:14.508 18602 ERROR nova.openstack.common.db.sqlalchemy.session [-] DB exception wrapped.

** Affects: nova
     Importance: High
         Status: Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1280034

Title:
  compute_node_update broken with havana compute nodes

Status in OpenStack Compute (Nova):
  Confirmed

Bug description:
  This change:

  https://review.openstack.org/#/c/66469

  Changed the format of the data in the "values" dictionary of
  compute_node_update. This causes an icehouse conductor to generate a
  broken SQL query when called from a havana compute node:

  http://logs.openstack.org/75/64075/13/check/check-grenade-
  dsvm/b70c839/logs/new/screen-n-cond.txt.gz?level=TRACE

  executors.base [-] Exception during message handling: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ': '2', u'io_workload': '0', u'num_instances': '2', u'num_vm_building': '0', u'nu' at line 1") 'UPDATE compute_nodes SET updated_at=%s, vcpus_used=%s, memory_mb_used=%s, free_ram_mb=%s, running_vms=%s, stats=%s WHERE compute_nodes.id = %s' (datetime.datetime(2014, 2, 12, 21, 2, 12, 395978), 4, 1216, 6737, 4, {u'num_task_None': 2, u'io_workload': 0, u'num_instances': 2, u'num_vm_active': 1, u'num_task_scheduling': 0, u'num_vm_building': 0, u'num_proj_d0e1e781676f4fe5b1b81e31b8ae87de': 1, u'num_vcpus_used': 2, u'num_proj_a8a2f9c3e3bd44edb1c5fd2ae4cc7b3c': 1, u'num_os_type_None': 2, u'num_vm_error': 1}, 1)
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base Traceback (most recent call last):
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/oslo.messaging/oslo/messaging/_executors/base.py", line 36, in _dispatch
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     incoming.reply(self.callback(incoming.ctxt, incoming.message))
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 134, in __call__
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     return self._dispatch(endpoint, method, ctxt, args)
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/conductor/manager.py", line 458, in compute_node_update
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     result = self.db.compute_node_update(context, node['id'], values)
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/db/api.py", line 228, in compute_node_update
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     return IMPL.compute_node_update(context, compute_id, values)
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 110, in wrapper
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     return f(*args, **kwargs)
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 166, in wrapped
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     return f(*args, **kwargs)
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 614, in compute_node_update
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     compute_ref.update(values)
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 456, in __exit__
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     self.commit()
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 368, in commit
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     self._prepare_impl()
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 347, in _prepare_impl
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     self.session.flush()
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base   File "/opt/stack/new/nova/nova/openstack/common/db/sqlalchemy/session.py", line 616, in _wrap
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base     raise exception.DBError(e)
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base DBError: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ': '2', u'io_workload': '0', u'num_instances': '2', u'num_vm_building': '0', u'nu' at line 1") 'UPDATE compute_nodes SET updated_at=%s, vcpus_used=%s, memory_mb_used=%s, free_ram_mb=%s, running_vms=%s, stats=%s WHERE compute_nodes.id = %s' (datetime.datetime(2014, 2, 12, 21, 2, 12, 395978), 4, 1216, 6737, 4, {u'num_task_None': 2, u'io_workload': 0, u'num_instances': 2, u'num_vm_active': 1, u'num_task_scheduling': 0, u'num_vm_building': 0, u'num_proj_d0e1e781676f4fe5b1b81e31b8ae87de': 1, u'num_vcpus_used': 2, u'num_proj_a8a2f9c3e3bd44edb1c5fd2ae4cc7b3c': 1, u'num_os_type_None': 2, u'num_vm_error': 1}, 1)
  2014-02-12 21:02:12.401 18598 TRACE oslo.messaging._executors.base 
  2014-02-12 21:02:12.404 18598 ERROR oslo.messaging._drivers.common [-] Returning exception (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ': '2', u'io_workload': '0', u'num_instances': '2', u'num_vm_building': '0', u'nu' at line 1") 'UPDATE compute_nodes SET updated_at=%s, vcpus_used=%s, memory_mb_used=%s, free_ram_mb=%s, running_vms=%s, stats=%s WHERE compute_nodes.id = %s' (datetime.datetime(2014, 2, 12, 21, 2, 12, 395978), 4, 1216, 6737, 4, {u'num_task_None': 2, u'io_workload': 0, u'num_instances': 2, u'num_vm_active': 1, u'num_task_scheduling': 0, u'num_vm_building': 0, u'num_proj_d0e1e781676f4fe5b1b81e31b8ae87de': 1, u'num_vcpus_used': 2, u'num_proj_a8a2f9c3e3bd44edb1c5fd2ae4cc7b3c': 1, u'num_os_type_None': 2, u'num_vm_error': 1}, 1) to caller
  2014-02-12 21:02:12.404 18598 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/_executors/base.py", line 36, in _dispatch\n    incoming.reply(self.callback(incoming.ctxt, incoming.message))\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 134, in __call__\n    return self._dispatch(endpoint, method, ctxt, args)\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/opt/stack/new/nova/nova/conductor/manager.py", line 458, in compute_node_update\n    result = self.db.compute_node_update(context, node[\'id\'], values)\n', '  File "/opt/stack/new/nova/nova/db/api.py", line 228, in compute_node_update\n    return IMPL.compute_node_update(context, compute_id, values)\n', '  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 110, in wrapper\n    return f(*args, **kwargs)\n', '  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 166, in wrapped\n    return f(*args, **kwargs)\n', '  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 614, in compute_node_update\n    compute_ref.update(values)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 456, in __exit__\n    self.commit()\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 368, in commit\n    self._prepare_impl()\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 347, in _prepare_impl\n    self.session.flush()\n', '  File "/opt/stack/new/nova/nova/openstack/common/db/sqlalchemy/session.py", line 616, in _wrap\n    raise exception.DBError(e)\n', 'DBError: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \': \'2\', u\'io_workload\': \'0\', u\'num_instances\': \'2\', u\'num_vm_building\': \'0\', u\'nu\' at line 1") \'UPDATE compute_nodes SET updated_at=%s, vcpus_used=%s, memory_mb_used=%s, free_ram_mb=%s, running_vms=%s, stats=%s WHERE compute_nodes.id = %s\' (datetime.datetime(2014, 2, 12, 21, 2, 12, 395978), 4, 1216, 6737, 4, {u\'num_task_None\': 2, u\'io_workload\': 0, u\'num_instances\': 2, u\'num_vm_active\': 1, u\'num_task_scheduling\': 0, u\'num_vm_building\': 0, u\'num_proj_d0e1e781676f4fe5b1b81e31b8ae87de\': 1, u\'num_vcpus_used\': 2, u\'num_proj_a8a2f9c3e3bd44edb1c5fd2ae4cc7b3c\': 1, u\'num_os_type_None\': 2, u\'num_vm_error\': 1}, 1)\n']
  2014-02-12 21:02:14.508 18602 ERROR nova.openstack.common.db.sqlalchemy.session [-] DB exception wrapped.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1280034/+subscriptions
Follow ups

[Bug 1280034] Re: compute_node_update broken with havana compute nodes
From: Thierry Carrez, 2014-03-05
[Bug 1280034] [NEW] compute_node_update broken with havana compute nodes
From: Dan Smith, 2014-02-14
References

[Bug 1280034] [NEW] compute_node_update broken with havana compute nodes
From: Dan Smith, 2014-02-14