yahoo-eng-team team mailing list archive

[Ryan <1759004@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Throughout OpenStack, the eventlet library is used heavily
for threading. In most cases, we use spawn_n() from
eventlet which does not return whether the method that was
wrapped with it raised an exception. Exceptions are instead
just printed to stderr. Many deployers are using simple
logging configurations that just log everything to a file.
This poses a problem because spawn_n() does not use standard
Python logging, but rather just prints a stack trace to stderr, like
this: https://github.com/eventlet/eventlet/blob/master/eventlet/greenpool.py#L93

Any exceptions that get thrown in enventlet.spawn_n() are
not shown in log files that are setup with normal log
configurations that write to files. Deployers who have not been
heavily involved with a lot of upstream development or do not
know the internals of eventlet would think that there are no
errors when tailing their log files which makes debugging
problems extremely difficult. There are a few options
to make error handling better

1. Use spawn() instead of spawn_n(). spawn() returns a greenthread
object that would allow us to see the results of the call. We could
check the results and log a proper exception if we used spawn().
The problem is, this would incur a performance penalty as spawn()
is slower than spawn_n()

2. Expect deployers to write a custom log handler that will redirect
stderr to their log file

3. Use proper exception handling inside of the functions that are called
with spawn_n() so that exceptions we care about get caught and logged
correctly

A specific case where this happens and causes a problem is here:
https://github.com/openstack/neutron/blob/7a722159f3bed10fbb26c4d0146f252da72ca0cd/neutron/services/segments/plugin.py#L222

In this scenario we reference aggregate.uuid. However if Nova API does not support
microversion 2.41, then this will raise an AttributeError. However, the AttributeError
will not be seen by most deployers because it will go to stderr rather than the log
files setup with the Python logging framework. Then we start to see strange behavior everywhere
and we have no idea why because our logs do not show anything. Logging and exception
handling needs to be spruced up in this case and others so that eventlet does
not devour important exceptions.

Ultimately, we need to be much more cautious of exception handling
inside of methods that are called through eventlet.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1759004

Title:
  No Errors or Warnings In Neutron When Nova API Microversions Are Not
  Supported

Status in neutron:
  New

Bug description:
  Throughout OpenStack, the eventlet library is used heavily
  for threading. In most cases, we use spawn_n() from
  eventlet which does not return whether the method that was
  wrapped with it raised an exception. Exceptions are instead
  just printed to stderr. Many deployers are using simple
  logging configurations that just log everything to a file.
  This poses a problem because spawn_n() does not use standard
  Python logging, but rather just prints a stack trace to stderr, like
  this: https://github.com/eventlet/eventlet/blob/master/eventlet/greenpool.py#L93

  Any exceptions that get thrown in enventlet.spawn_n() are
  not shown in log files that are setup with normal log
  configurations that write to files. Deployers who have not been
  heavily involved with a lot of upstream development or do not
  know the internals of eventlet would think that there are no
  errors when tailing their log files which makes debugging
  problems extremely difficult. There are a few options
  to make error handling better

  1. Use spawn() instead of spawn_n(). spawn() returns a greenthread
  object that would allow us to see the results of the call. We could
  check the results and log a proper exception if we used spawn().
  The problem is, this would incur a performance penalty as spawn()
  is slower than spawn_n()

  2. Expect deployers to write a custom log handler that will redirect
  stderr to their log file

  3. Use proper exception handling inside of the functions that are called
  with spawn_n() so that exceptions we care about get caught and logged
  correctly

  A specific case where this happens and causes a problem is here:
  https://github.com/openstack/neutron/blob/7a722159f3bed10fbb26c4d0146f252da72ca0cd/neutron/services/segments/plugin.py#L222

  In this scenario we reference aggregate.uuid. However if Nova API does not support
  microversion 2.41, then this will raise an AttributeError. However, the AttributeError
  will not be seen by most deployers because it will go to stderr rather than the log
  files setup with the Python logging framework. Then we start to see strange behavior everywhere
  and we have no idea why because our logs do not show anything. Logging and exception
  handling needs to be spruced up in this case and others so that eventlet does
  not devour important exceptions.

  Ultimately, we need to be much more cautious of exception handling
  inside of methods that are called through eventlet.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1759004/+subscriptions