yahoo-eng-team team mailing list archive

[Ilya Shakhat <shakhat@xxxxxxxxx>]

In OSProfiler the current state is stored in thread-local variable. The
definition of variable is in
https://git.openstack.org/cgit/openstack/osprofiler/tree/osprofiler/profiler.py#n30.

It is observed however that the variable remains the same when a new
green thread is forked. As result code
https://git.openstack.org/cgit/openstack/nova/tree/nova/utils.py#n905
resets the variable of the process itself, not the one belonging to a
thread.

Steps to observe:
1. Print type and address of variable __local_ctx in osprofiler.profiler functions _Profiler.start and _Profiler.stop
The output shows that the type of variable is 'thread._local' and address is the same across all invocations of Profiler's start and stop function.
  

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1743586

Title:
  Nova operations fail when profiling is enabled

Status in OpenStack Compute (nova):
  New
Status in osprofiler:
  New

Bug description:
  Nova server-related operations fail with HTTP 500 when executed with
  profiling. Nova server log contains error "IndexError: pop from an
  empty deque" and stack trace pointing to osprofiler code:

  ERROR nova.api.openstack   File "/opt/stack/osprofiler/osprofiler/web.py", line 132, in __call__
  ERROR nova.api.openstack     return request.get_response(self.application)
  ERROR nova.api.openstack   File "/opt/stack/osprofiler/osprofiler/profiler.py", line 348, in __exit__
  ERROR nova.api.openstack     stop()
  ERROR nova.api.openstack   File "/opt/stack/osprofiler/osprofiler/profiler.py", line 87, in stop
  ERROR nova.api.openstack     profiler.stop(info=info)
  ERROR nova.api.openstack   File "/opt/stack/osprofiler/osprofiler/profiler.py", line 406, in stop
  ERROR nova.api.openstack     self._notify("%s-stop" % self._name.pop(), info)

  
  Steps to reproduce:
   0. Take the latest Devstack with osprofiler enabled.
   1. Execute `openstack --os-profile SECRET_KEY server list`

  Observed behavior:
    $ openstack --os-profile SECRET_KEY server list
    The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-73de8073-94b7-44f0-96ca-5a0c5979636c)
    Trace ID: 5f0d7147-6e15-41c4-a61b-84f74e3787d4
    Display trace with command:
    osprofiler trace show --html 5f0d7147-6e15-41c4-a61b-84f74e3787d4

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1743586/+subscriptions