yahoo-eng-team team mailing list archive

[Abhishek Kekane <1978971@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Earlier when we added config reloading functionality we made provision
for the workers who were performing existing tasks should complete those
before terminating them. The sequence was likely below;

On receipt of a SIGHUP signal the master process will:

* reload the configuration
* send a SIGHUP to the original workers
* start (a potentially different number of) new workers with the new
  configuration
* its listening socket will *not* be closed

On receipt of a SIGHUP signal each original worker process will:

* close the listening socket so as not to accept new requests
* complete any in-flight requests
* complete async requests (V1 create with copy-from option and V2 task api)
* exit

Recently while working on one of the functionality I found that the
workers are not waiting to complete any in-flight requests due to some
errors.


Following are some important logs;
[1] Logs for upload/import request which gets terminated on reload/sighup
[2] Reproducer

[1] https://paste.opendev.org/show/bTjORjtLtbhUe93xQFDu/
[2] https://paste.opendev.org/show/bFqCD4U4P0Q8MyIrsO4J/

** Affects: glance
     Importance: High
         Status: New

** Changed in: glance
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1978971

Title:
  Graceful shutdown not working for wsgi and uwsgi

Status in Glance:
  New

Bug description:
  Earlier when we added config reloading functionality we made provision
  for the workers who were performing existing tasks should complete
  those before terminating them. The sequence was likely below;

  On receipt of a SIGHUP signal the master process will:

  * reload the configuration
  * send a SIGHUP to the original workers
  * start (a potentially different number of) new workers with the new
    configuration
  * its listening socket will *not* be closed

  On receipt of a SIGHUP signal each original worker process will:

  * close the listening socket so as not to accept new requests
  * complete any in-flight requests
  * complete async requests (V1 create with copy-from option and V2 task api)
  * exit

  Recently while working on one of the functionality I found that the
  workers are not waiting to complete any in-flight requests due to some
  errors.

  
  Following are some important logs;
  [1] Logs for upload/import request which gets terminated on reload/sighup
  [2] Reproducer

  [1] https://paste.opendev.org/show/bTjORjtLtbhUe93xQFDu/
  [2] https://paste.opendev.org/show/bFqCD4U4P0Q8MyIrsO4J/

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1978971/+subscriptions