yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1621987] [NEW] Power state synchronization in compute is too agressive

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mathieu Gagné <1621987@xxxxxxxxxxxxxxxxxx>
Date: Fri, 09 Sep 2016 19:32:39 -0000
Reply-to: Bug 1621987 <1621987@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

By default, 1000 greenthreads can be used to synchronize the power state
of instances running on a compute node.

In the Ironic context, this means 1000 simultaneous HTTP requests can be
initiated to the Ironic API. Some of those requests can fail with the
following error:

    ERROR nova.compute.manager [-] [instance: XXXXXXXXXXXXXX] Periodic sync_power_state task had an error while processing an instance.
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] Traceback (most recent call last):
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 6083, in _sync
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     query_driver_power_state_and_sync()
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 252, in inner
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return f(*args, **kwargs)
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 6080, in query_driver_power_state_and_sync
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     self._query_driver_power_state_and_sync(context, db_instance)
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 6110, in _query_driver_power_state_and_sync
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     vm_instance = self.driver.get_info(db_instance)
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 557, in get_info
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     node = _validate_instance_and_node(self.ironicclient, instance)
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 126, in _validate_instance_and_node
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return ironicclient.call("node.get_by_instance_uuid", instance.uuid)
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/client_wrapper.py", line 122, in call
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return self._multi_getattr(client, method)(*args, **kwargs)
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/v1/node.py", line 151, in get_by_instance_uuid
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     nodes = self._list(self._path(path), 'nodes')
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/base.py", line 119, in _list
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     resp, body = self.api.json_request('GET', url)
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 351, in json_request
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     resp, body_iter = self._http_request(url, method, **kwargs)
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 160, in wrapper
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return func(self, url, method, **kwargs)
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 296, in _http_request
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     resp = conn.getresponse()
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     response.begin()
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/usr/lib/python2.7/httplib.py", line 415, in begin
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     version, status, reason = self._read_status()
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/usr/lib/python2.7/httplib.py", line 379, in _read_status
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     raise BadStatusLine(line)
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] BadStatusLine: ''
    ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] 

There should be a way to limit the number of simultaneous requests made
against the hypervisor to synchronize the instance power states so it's
not overwhelm.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1621987

Title:
  Power state synchronization in compute is too agressive

Status in OpenStack Compute (nova):
  New

Bug description:
  By default, 1000 greenthreads can be used to synchronize the power
  state of instances running on a compute node.

  In the Ironic context, this means 1000 simultaneous HTTP requests can
  be initiated to the Ironic API. Some of those requests can fail with
  the following error:

      ERROR nova.compute.manager [-] [instance: XXXXXXXXXXXXXX] Periodic sync_power_state task had an error while processing an instance.
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] Traceback (most recent call last):
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 6083, in _sync
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     query_driver_power_state_and_sync()
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 252, in inner
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return f(*args, **kwargs)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 6080, in query_driver_power_state_and_sync
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     self._query_driver_power_state_and_sync(context, db_instance)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 6110, in _query_driver_power_state_and_sync
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     vm_instance = self.driver.get_info(db_instance)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 557, in get_info
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     node = _validate_instance_and_node(self.ironicclient, instance)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 126, in _validate_instance_and_node
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return ironicclient.call("node.get_by_instance_uuid", instance.uuid)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/client_wrapper.py", line 122, in call
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return self._multi_getattr(client, method)(*args, **kwargs)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/v1/node.py", line 151, in get_by_instance_uuid
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     nodes = self._list(self._path(path), 'nodes')
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/base.py", line 119, in _list
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     resp, body = self.api.json_request('GET', url)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 351, in json_request
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     resp, body_iter = self._http_request(url, method, **kwargs)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 160, in wrapper
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return func(self, url, method, **kwargs)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 296, in _http_request
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     resp = conn.getresponse()
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     response.begin()
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/usr/lib/python2.7/httplib.py", line 415, in begin
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     version, status, reason = self._read_status()
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/usr/lib/python2.7/httplib.py", line 379, in _read_status
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     raise BadStatusLine(line)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] BadStatusLine: ''
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] 

  There should be a way to limit the number of simultaneous requests
  made against the hypervisor to synchronize the instance power states
  so it's not overwhelm.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1621987/+subscriptions
Follow ups

[Bug 1621987] Re: Power state synchronization in compute is too agressive
From: OpenStack Infra, 2016-09-12