yahoo-eng-team team mailing list archive

[OpenStack Infra <1621987@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/367746
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=386812e287198cd2d340d273753ef06075f7c05d
Submitter: Jenkins
Branch:    master

commit 386812e287198cd2d340d273753ef06075f7c05d
Author: Mathieu Gagné <mgagne@xxxxxxxx>
Date:   Fri Sep 9 00:35:50 2016 -0400

    Add sync_power_state_pool_size option
    
    The sync_power_state_pool_size option allows to set
    the number of greenthreads available for use to sync power states.
    
    It can be used to reduce the number of concurrent requests
    made to the hypervisor or system with real instance power states
    for performance reasons.
    
    Closes-bug: #1621987
    Change-Id: I9cf900314d71c44ec51eb190c102ddc60e13a767


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1621987

Title:
  Power state synchronization in compute is too agressive

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  By default, 1000 greenthreads can be used to synchronize the power
  state of instances running on a compute node.

  In the Ironic context, this means 1000 simultaneous HTTP requests can
  be initiated to the Ironic API. Some of those requests can fail with
  the following error:

      ERROR nova.compute.manager [-] [instance: XXXXXXXXXXXXXX] Periodic sync_power_state task had an error while processing an instance.
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] Traceback (most recent call last):
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 6083, in _sync
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     query_driver_power_state_and_sync()
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 252, in inner
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return f(*args, **kwargs)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 6080, in query_driver_power_state_and_sync
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     self._query_driver_power_state_and_sync(context, db_instance)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 6110, in _query_driver_power_state_and_sync
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     vm_instance = self.driver.get_info(db_instance)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 557, in get_info
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     node = _validate_instance_and_node(self.ironicclient, instance)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 126, in _validate_instance_and_node
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return ironicclient.call("node.get_by_instance_uuid", instance.uuid)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/client_wrapper.py", line 122, in call
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return self._multi_getattr(client, method)(*args, **kwargs)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/v1/node.py", line 151, in get_by_instance_uuid
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     nodes = self._list(self._path(path), 'nodes')
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/base.py", line 119, in _list
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     resp, body = self.api.json_request('GET', url)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 351, in json_request
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     resp, body_iter = self._http_request(url, method, **kwargs)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 160, in wrapper
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     return func(self, url, method, **kwargs)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line 296, in _http_request
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     resp = conn.getresponse()
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     response.begin()
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/usr/lib/python2.7/httplib.py", line 415, in begin
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     version, status, reason = self._read_status()
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]   File "/usr/lib/python2.7/httplib.py", line 379, in _read_status
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]     raise BadStatusLine(line)
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] BadStatusLine: ''
      ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] 

  There should be a way to limit the number of simultaneous requests
  made against the hypervisor to synchronize the instance power states
  so it's not overwhelm.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1621987/+subscriptions