yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1575106] [NEW] Keystone crash with coredump during apache stop

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Michał Górniak <mgorniak@xxxxxxxxxxxx>
Date: Tue, 26 Apr 2016 10:35:03 -0000
Reply-to: Bug 1575106 <1575106@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hey,
i have lots of coredump from keystone during apache stop/restart

~# keystone --version
1.3.1

(gdb) where
#0  0x00007f32631c7730 in _PyTrash_thread_destroy_chain () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#1  0x00007f326317ff4e in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#2  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3  0x00007f326317fdd8 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f326317fdd8 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f32631b67a5 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f326317c3b1 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f3263180059 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f3263180059 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f32631b66d0 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f32630ae7bd in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f326319b577 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f3263103f92 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f3269a8f182 in start_thread (arg=0x7f32414e3700) at pthread_create.c:312
#20 0x00007f32697bc47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


(gdb) bt full
#0  0x00007f32631c7730 in _PyTrash_thread_destroy_chain () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#1  0x00007f326317ff4e in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#2  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#3  0x00007f326317fdd8 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#4  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#5  0x00007f326317fdd8 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#6  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#7  0x00007f32631b67a5 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#8  0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#9  0x00007f326317c3b1 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#10 0x00007f3263180059 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#11 0x00007f3263180059 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#12 0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#13 0x00007f32631b66d0 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#14 0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#15 0x00007f32630ae7bd in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#16 0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#17 0x00007f326319b577 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#18 0x00007f3263103f92 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
No symbol table info available.
#19 0x00007f3269a8f182 in start_thread (arg=0x7f32414e3700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7f32414e3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139853820737280, -3495166348351394815, 0, 0, 139853820737984, 139853820737280, 3538191194546613249, 3538281094028176385}, mask_was_saved = 0}}, priv = {
            pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#20 0x00007f32697bc47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


its looking similar to:
http://bugs.python.org/issue1856

output from  thread_exit.py can be found here:
http://pastebin.com/f0vAH1UK

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1575106

Title:
  Keystone crash with coredump during apache stop

Status in OpenStack Identity (keystone):
  New

Bug description:
  Hey,
  i have lots of coredump from keystone during apache stop/restart

  ~# keystone --version
  1.3.1

  (gdb) where
  #0  0x00007f32631c7730 in _PyTrash_thread_destroy_chain () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #1  0x00007f326317ff4e in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #2  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #3  0x00007f326317fdd8 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #4  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #5  0x00007f326317fdd8 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #6  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #7  0x00007f32631b67a5 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #8  0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #9  0x00007f326317c3b1 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #10 0x00007f3263180059 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #11 0x00007f3263180059 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #12 0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #13 0x00007f32631b66d0 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #14 0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #15 0x00007f32630ae7bd in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #16 0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #17 0x00007f326319b577 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #18 0x00007f3263103f92 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  #19 0x00007f3269a8f182 in start_thread (arg=0x7f32414e3700) at pthread_create.c:312
  #20 0x00007f32697bc47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


  (gdb) bt full
  #0  0x00007f32631c7730 in _PyTrash_thread_destroy_chain () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #1  0x00007f326317ff4e in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #2  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #3  0x00007f326317fdd8 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #4  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #5  0x00007f326317fdd8 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #6  0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #7  0x00007f32631b67a5 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #8  0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #9  0x00007f326317c3b1 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #10 0x00007f3263180059 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #11 0x00007f3263180059 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #12 0x00007f326318154d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #13 0x00007f32631b66d0 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #14 0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #15 0x00007f32630ae7bd in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #16 0x00007f3263122d43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #17 0x00007f326319b577 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #18 0x00007f3263103f92 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
  No symbol table info available.
  #19 0x00007f3269a8f182 in start_thread (arg=0x7f32414e3700) at pthread_create.c:312
          __res = <optimized out>
          pd = 0x7f32414e3700
          now = <optimized out>
          unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139853820737280, -3495166348351394815, 0, 0, 139853820737984, 139853820737280, 3538191194546613249, 3538281094028176385}, mask_was_saved = 0}}, priv = {
              pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
          not_first_call = <optimized out>
          pagesize_m1 = <optimized out>
          sp = <optimized out>
          freesize = <optimized out>
          __PRETTY_FUNCTION__ = "start_thread"
  #20 0x00007f32697bc47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

  
  its looking similar to:
  http://bugs.python.org/issue1856

  output from  thread_exit.py can be found here:
  http://pastebin.com/f0vAH1UK

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1575106/+subscriptions
Follow ups

[Bug 1575106] Re: Keystone crash with coredump during apache stop
From: Serge Kovaleff, 2016-05-04