yahoo-eng-team team mailing list archive

[Brian Haley <1759004@xxxxxxxxxxxxxxxxxx>]

Moving to Resolved since a change was merged to fix the known issue,
please feel free to reopen if needed.

** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1759004

Title:
  No Errors or Warnings In Neutron When Nova API Microversions Are Not
  Supported

Status in neutron:
  Fix Released

Bug description:
  Throughout OpenStack, the eventlet library is used heavily
  for threading. In most cases, we use spawn_n() from
  eventlet which does not return whether the method that was
  wrapped with it raised an exception. Exceptions are instead
  just printed to stderr. Many deployers are using simple
  logging configurations that just log everything to a file.
  This poses a problem because spawn_n() does not use standard
  Python logging, but rather just prints a stack trace to stderr, like
  this: https://github.com/eventlet/eventlet/blob/master/eventlet/greenpool.py#L93

  Any exceptions that get thrown in enventlet.spawn_n() are
  not shown in log files that are setup with normal log
  configurations that write to files. Deployers who have not been
  heavily involved with a lot of upstream development or do not
  know the internals of eventlet would think that there are no
  errors when tailing their log files which makes debugging
  problems extremely difficult. There are a few options
  to make error handling better

  1. Use spawn() instead of spawn_n(). spawn() returns a greenthread
  object that would allow us to see the results of the call. We could
  check the results and log a proper exception if we used spawn().
  The problem is, this would incur a performance penalty as spawn()
  is slower than spawn_n()

  2. Expect deployers to write a custom log handler that will redirect
  stderr to their log file

  3. Use proper exception handling inside of the functions that are called
  with spawn_n() so that exceptions we care about get caught and logged
  correctly

  A specific case where this happens and causes a problem is here:
  https://github.com/openstack/neutron/blob/7a722159f3bed10fbb26c4d0146f252da72ca0cd/neutron/services/segments/plugin.py#L222

  In this scenario we reference aggregate.uuid. However if Nova API does not support microversion 2.41, then this will raise an AttributeError. However, the AttributeError will not be seen by most deployers because it will go to stderr rather than the log files setup with the Python logging framework. Then we start to see strange behavior everywhere
  and we have no idea why because our logs do not show anything. Logging and exception handling needs to be spruced up in this case and others so that eventlet does not devour important exceptions.

  Ultimately, we need to be much more cautious of exception handling
  inside of methods that are called through eventlet.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1759004/+subscriptions