yahoo-eng-team team mailing list archive

[Tiantian Gao <gtt116@xxxxxxxxx>]

Public bug reported:

In IceHouse, An awesome feature got implemented: graceful shutdown nova
service, which can make sure in-process RPC request got done before kill
the process.

But nova-api not support graceful shutdown now, which can cause problem
when do upgrading. For example, when a request to create an instance was
in-progress, kill the nova-api may lead to quota not sync or odd
database records. Especially in large-scale development, there are
hundreds of request in a second, kill the nova-api will interrupt lots
in-process greenlet.

In nova/wsgi.py, when stoping WSGI service, we first shrink the greenlet
pool size to 0, then kill the eventlet wsgi server. The work around is
quick and easy: wait for all greenlets in the pool to finish, then kill
wsgi server. The code looks like below:


diff --git a/nova/wsgi.py b/nova/wsgi.py
index ba52872..3c89297 100644
--- a/nova/wsgi.py
+++ b/nova/wsgi.py
@@ -212,6 +212,9 @@ class Server(object):
         if self._server is not None:
             # Resize pool to stop new requests from being processed
             self._pool.resize(0)
+            num = self._pool.running()
+            LOG.info(_("Waiting WSGI server to finish %d requests." % num))
+            self._pool.waitall()
             self._server.kill()
 
     def wait(self):

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1382390

Title:
  nova-api should shutdown gracefully

Status in OpenStack Compute (Nova):
  New

Bug description:
  In IceHouse, An awesome feature got implemented: graceful shutdown
  nova service, which can make sure in-process RPC request got done
  before kill the process.

  But nova-api not support graceful shutdown now, which can cause
  problem when do upgrading. For example, when a request to create an
  instance was in-progress, kill the nova-api may lead to quota not sync
  or odd database records. Especially in large-scale development, there
  are hundreds of request in a second, kill the nova-api will interrupt
  lots in-process greenlet.

  In nova/wsgi.py, when stoping WSGI service, we first shrink the
  greenlet pool size to 0, then kill the eventlet wsgi server. The work
  around is quick and easy: wait for all greenlets in the pool to
  finish, then kill wsgi server. The code looks like below:

  
  diff --git a/nova/wsgi.py b/nova/wsgi.py
  index ba52872..3c89297 100644
  --- a/nova/wsgi.py
  +++ b/nova/wsgi.py
  @@ -212,6 +212,9 @@ class Server(object):
           if self._server is not None:
               # Resize pool to stop new requests from being processed
               self._pool.resize(0)
  +            num = self._pool.running()
  +            LOG.info(_("Waiting WSGI server to finish %d requests." % num))
  +            self._pool.waitall()
               self._server.kill()
   
       def wait(self):

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1382390/+subscriptions