yahoo-eng-team team mailing list archive

[OpenStack Infra <1737214@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/526748
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7c30da13842469e641d171b6008a57d5db8c5d5e
Submitter: Zuul
Branch:    master

commit 7c30da13842469e641d171b6008a57d5db8c5d5e
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Fri Dec 8 13:15:50 2017 -0500

    Don't launch guestfs in a thread pool if guestfs.debug is enabled
    
    When guestfs.debug is enabled, we're handling callback events
    from guestfs and logging them at debug level. When guestfs
    is launched to inspect capabilities, that is currently done
    in an eventlet thread pool. Because of the concurrent logging
    along with the eventlet thread, we can hit an issue where eventlet
    tries to switch threads and fails and then we hang the launch
    call to guestfs, which hangs creating an instance.
    
    This change simply avoids using a thread pool to launch guestfs
    if guestfs.debug is True.
    
    Change-Id: I0ffe93a031154b123c8beff96a695df5a280b935
    Closes-Bug: #1737214


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1737214

Title:
  Enabling guestfs.debug can hang guestfs.launch

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Confirmed
Status in OpenStack Compute (nova) ocata series:
  Confirmed
Status in OpenStack Compute (nova) pike series:
  Confirmed

Bug description:
  I noticed this while debugging this change:

  https://review.openstack.org/#/c/524710/

  When we enable guestfs.debug, we're receiving events from guestfs and
  logging them.

  However, guestfs.launch is run in an eventlet thread pool, and running
  things within an eventlet thread pool for native apps is a no-no for
  doing logging because it can cause a greenthread switch that makes us
  lose the return from the guestfs.launch call, which eventually means
  we hang when trying to create the instance.

  So we need to just run guestfs.launch non-threaded if guestfs.debug is
  enabled.

  From the logs, you can see the guestfs.launch thread switch here:

  http://logs.openstack.org/50/524750/1/check/legacy-tempest-dsvm-
  neutron-full-
  centos-7/a7f051e/logs/screen-n-cpu.txt.gz#_Dec_04_13_43_19_941869

  And the logging callback here:

  http://logs.openstack.org/50/524750/1/check/legacy-tempest-dsvm-
  neutron-full-
  centos-7/a7f051e/logs/screen-n-cpu.txt.gz#_Dec_04_15_25_29_320753

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1737214/+subscriptions