yahoo-eng-team team mailing list archive

[Mitsuhiro Tanino <mitsuhiro.tanino@xxxxxxx>]

** Also affects: oslo.service
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1382390

Title:
  nova-api should shutdown gracefully

Status in OpenStack Compute (Nova):
  In Progress
Status in The Oslo library incubator:
  Confirmed
Status in Library for running OpenStack services:
  New

Bug description:
  In IceHouse, An awesome feature got implemented: graceful shutdown
  nova service, which can make sure in-process RPC request got done
  before kill the process.

  But nova-api not support graceful shutdown now, which can cause
  problem when do upgrading. For example, when a request to create an
  instance was in-progress, kill the nova-api may lead to quota not sync
  or odd database records. Especially in large-scale development, there
  are hundreds of request in a second, kill the nova-api will interrupt
  lots in-process greenlet.

  In nova/wsgi.py, when stoping WSGI service, we first shrink the
  greenlet pool size to 0, then kill the eventlet wsgi server. The work
  around is quick and easy: wait for all greenlets in the pool to
  finish, then kill wsgi server. The code looks like below:

  
  diff --git a/nova/wsgi.py b/nova/wsgi.py
  index ba52872..3c89297 100644
  --- a/nova/wsgi.py
  +++ b/nova/wsgi.py
  @@ -212,6 +212,9 @@ class Server(object):
           if self._server is not None:
               # Resize pool to stop new requests from being processed
               self._pool.resize(0)
  +            num = self._pool.running()
  +            LOG.info(_("Waiting WSGI server to finish %d requests." % num))
  +            self._pool.waitall()
               self._server.kill()
   
       def wait(self):

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1382390/+subscriptions