yahoo-eng-team team mailing list archive

[OpenStack Infra <1825584@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/724188
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d6f664524dfa4eecbe3e16fda0fec5359faf84c6
Submitter: Zuul
Branch:    master

commit d6f664524dfa4eecbe3e16fda0fec5359faf84c6
Author: Sean Mooney <work@xxxxxxxxxxxxxxx>
Date:   Tue Apr 28 23:20:16 2020 +0100

    Silence amqp heartbeat warning
    
    When the nova api is executing under uWSGI or MOD_WSGI
    the lifetime of the amqp heartbeat thread is controlled
    by the wsgi server. As a result when the nova api is run
    in this configuration we expect that the heartbeat thread
    will be suspended and heartbeats will be missed when the wsgi
    server suspends execution of the wsgi application.
    
    This change adds a python logging filter to suppress
    the reporting of heartbeat warnings as this behavior is
    expected. Since the operator cannot do anything to address
    the issue the warning is just noise and many operators
    and customers find it to be off-putting.
    
    Change-Id: I642b1e3ed6de2be4dcc19fe214f84095d2e1d31a
    Closes-Bug: #1825584


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1825584

Title:
  eventlet monkey-patching breaks AMQP heartbeat on uWSGI

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Stein nova-api running under uWSGI presents an AMQP issue. The first
  API call that requires RPC creates an AMQP connection and successfully
  completes. Normally regular heartbeats would be sent from this point
  on, to maintain the connection. This is not happening. After a few
  minutes, the AMQP server (rabbitmq, in my case) notices that there
  have been no heartbeats, and drops the connection. A later nova API
  call that requires RPC tries to use the old connection, and throws a
  "connection reset by peer" exception and the API call fails. A
  mailing-list response suggests that this is affecting mod_wsgi also:

  http://lists.openstack.org/pipermail/openstack-
  discuss/2019-April/005310.html

  I've discovered that this problem seems to be caused by eventlet
  monkey-patching, which was introduced in:

  https://github.com/openstack/nova/commit/23ba1c690652832c655d57476630f02c268c87ae

  It was later rearranged in:

  https://github.com/openstack/nova/commit/3c5e2b0e9fac985294a949852bb8c83d4ed77e04

  but this problem remains.

  If I comment out the import of nova.monkey_patch in
  nova/api/openstack/__init__.py the problem goes away.

  Seems that eventlet monkey-patching and uWSGI are not getting along
  for some reason...

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1825584/+subscriptions