← Back to team overview

group.of.nepali.translators team mailing list archive

  1. group.of.nepali.translators team
  2. Mailing list archive
  3. Message #10398

[Bug 1597411] Re: host move inside CRUSH cause permanent crash for all mons

  Thread Previous • Date Previous • • Thread Next 
** Changed in: ceph (Ubuntu Zesty)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1597411

Title:
  host move inside CRUSH cause permanent crash for all mons

Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Xenial:
  Triaged
Status in ceph source package in Yakkety:
  Triaged
Status in ceph source package in Zesty:
  Fix Released

Bug description:
  Upstream bugreport: http://tracker.ceph.com/issues/16525

  affected version: 10.2.0-0ubuntu0.16.04.2

  Copy form upstream:

  
  I've got crashed all ceph mons simultaneous when I've tried to move host with working OSD from one root to the other.

  My command was:

  ceph osd crush move pp1 root=fast2500

  ```
  Trace:
  9> 2016-06-29 14:10:30.337919 7f6e3951d700 5 - op tracker -- seq: 11, time: 2016-06-29 14:10:30.337919, event: callback finished, op: mon_command({"prefix": "osd crush move", "args": ["root=fast2500"], "name": "pp1"} v 0)
  8> 2016-06-29 14:10:30.337924 7f6e3951d700 5 - op tracker -- seq: 11, time: 2016-06-29 14:10:30.337924, event: psvc:dispatch, op: mon_command({"prefix": "osd crush move", "args": ["root=fast2500"], "name": "pp1"} v 0)
  7> 2016-06-29 14:10:30.337927 7f6e3951d700 5 mon.pp5@0(leader).paxos(paxos active c 2446247..2446981) is_readable = 1 - now=2016-06-29 14:10:30.337928 lease_expire=2016-06-29 14:10:35.337162 has v0 lc 2446981
  -6> 2016-06-29 14:10:30.337950 7f6e3951d700 5 - op tracker -- seq: 11, time: 2016-06-29 14:10:30.337950, event: osdmap:preprocess_query, op: mon_command({"prefix": "osd crush move", "args": ["root=fast2500"], "name": "pp1"} v 0)
  5> 2016-06-29 14:10:30.337956 7f6e3951d700 5 - op tracker -- seq: 11, time: 2016-06-29 14:10:30.337956, event: osdmap:preprocess_command, op: mon_command({"prefix": "osd crush move", "args": ["root=fast2500"], "name": "pp1"} v 0)
  4> 2016-06-29 14:10:30.338007 7f6e3951d700 5 - op tracker -- seq: 11, time: 2016-06-29 14:10:30.338007, event: osdmap:prepare_update, op: mon_command({"prefix": "osd crush move", "args": ["root=fast2500"], "name": "pp1"} v 0)
  3> 2016-06-29 14:10:30.338016 7f6e3951d700 5 - op tracker -- seq: 11, time: 2016-06-29 14:10:30.338015, event: osdmap:prepare_command, op: mon_command({"prefix": "osd crush move", "args": ["root=fast2500"], "name": "pp1"} v 0)
  2> 2016-06-29 14:10:30.338039 7f6e3951d700 5 - op tracker -- seq: 11, time: 2016-06-29 14:10:30.338036, event: osdmap:prepare_command_impl, op: mon_command({"prefix": "osd crush move", "args": ["root=fast2500"], "name": "pp1"} v 0)
  -1> 2016-06-29 14:10:30.338052 7f6e3951d700 0 mon.pp5@0(leader).osd e10230 moving crush item name 'pp1' to location {root=fast2500}
  0> 2016-06-29 14:10:30.341861 7f6e3951d700 -1 crush/CrushWrapper.h: In function 'int CrushWrapper::detach_bucket(CephContext*, int)' thread 7f6e3951d700 time 2016-06-29 14:10:30.338135
  crush/CrushWrapper.h: 940: FAILED assert(successful_detach)

  ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55cda5fb9fa0]
   2: (()+0x560833) [0x55cda5ec8833]
   3: (CrushWrapper::move_bucket(CephContext*, int, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&)+0xda) [0x55cda5ec644a]
   4: (OSDMonitor::prepare_command_impl(std::shared_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > >&)+0x2cfe) [0x55cda5c8701e]
   5: (OSDMonitor::prepare_command(std::shared_ptr<MonOpRequest>)+0x2ff) [0x55cda5c9903f]
   6: (OSDMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x24b) [0x55cda5c9958b]
   7: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xb4f) [0x55cda5c4c0af]
   8: (PaxosService::C_RetryMessage::_finish(int)+0x58) [0x55cda5c4d698]
   9: (C_MonOp::finish(int)+0x82) [0x55cda5c15862]
   10: (Context::complete(int)+0x9) [0x55cda5c14949]
   11: (void finish_contexts<Context>(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0x1fb) [0x55cda5c1b25b]
   12: (Paxos::finish_round()+0x287) [0x55cda5c41b17]
   13: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xe19) [0x55cda5c42cf9]
   14: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x250) [0x55cda5c43520]
   15: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xa38) [0x55cda5c0ee68]
   16: (Monitor::_ms_dispatch(Message*)+0x554) [0x55cda5c0f664]
   17: (Monitor::ms_dispatch(Message*)+0x23) [0x55cda5c326f3]
   18: (DispatchQueue::entry()+0xf2b) [0x55cda60aedfb]
   19: (DispatchQueue::DispatchThread::entry()+0xd) [0x55cda5fa032d]
   20: (()+0x76fa) [0x7f6e4165a6fa]
   21: (clone()+0x6d) [0x7f6e3f916b5d]
   NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

  --- logging levels ---
  0/ 5 none
  0/ 1 lockdep
  0/ 1 context
  1/ 1 crush
  1/ 5 mds
  1/ 5 mds_balancer
  1/ 5 mds_locker
  1/ 5 mds_log
  1/ 5 mds_log_expire
  1/ 5 mds_migrator
  0/ 1 buffer
  0/ 1 timer
  0/ 1 filer
  0/ 1 striper
  0/ 1 objecter
  0/ 5 rados
  0/ 5 rbd
  0/ 5 rbd_mirror
  0/ 5 rbd_replay
  0/ 5 journaler
  0/ 5 objectcacher
  0/ 5 client
  0/ 5 osd
  0/ 5 optracker
  0/ 5 objclass
  1/ 3 filestore
  1/ 3 journal
  0/ 5 ms
  1/ 5 mon
  0/10 monc
  1/ 5 paxos
  0/ 5 tp
  1/ 5 auth
  1/ 5 crypto
  1/ 1 finisher
  1/ 5 heartbeatmap
  1/ 5 perfcounter
  1/ 5 rgw
  1/10 civetweb
  1/ 5 javaclient
  1/ 5 asok
  1/ 1 throttle
  0/ 0 refs
  1/ 5 xio
  1/ 5 compressor
  1/ 5 newstore
  1/ 5 bluestore
  1/ 5 bluefs
  1/ 3 bdev
  1/ 5 kstore
  4/ 5 rocksdb
  4/ 5 leveldb
  1/ 5 kinetic
  1/ 5 fuse
  2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent 10000
  max_new 1000
  log_file /var/log/ceph/ceph-mon.pp5.log
  -- end dump of recent events ---
  2016-06-29 14:10:30.346791 7f6e3951d700 -1 ** Caught signal (Aborted) *
  in thread 7f6e3951d700 thread_name:ms_dispatch

  ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
   1: (()+0x5233be) [0x55cda5e8b3be]
   2: (()+0x113d0) [0x7f6e416643d0]
   3: (gsignal()+0x38) [0x7f6e3f845418]
   4: (abort()+0x16a) [0x7f6e3f84701a]
   5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55cda5fba18b]
   6: (()+0x560833) [0x55cda5ec8833]
   7: (CrushWrapper::move_bucket(CephContext*, int, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&)+0xda) [0x55cda5ec644a]
   8: (OSDMonitor::prepare_command_impl(std::shared_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > >&)+0x2cfe) [0x55cda5c8701e]
   9: (OSDMonitor::prepare_command(std::shared_ptr<MonOpRequest>)+0x2ff) [0x55cda5c9903f]
   10: (OSDMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x24b) [0x55cda5c9958b]
   11: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xb4f) [0x55cda5c4c0af]
   12: (PaxosService::C_RetryMessage::_finish(int)+0x58) [0x55cda5c4d698]
   13: (C_MonOp::finish(int)+0x82) [0x55cda5c15862]
   14: (Context::complete(int)+0x9) [0x55cda5c14949]
   15: (void finish_contexts<Context>(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0x1fb) [0x55cda5c1b25b]
   16: (Paxos::finish_round()+0x287) [0x55cda5c41b17]
   17: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xe19) [0x55cda5c42cf9]
   18: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x250) [0x55cda5c43520]
   19: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xa38) [0x55cda5c0ee68]
   20: (Monitor::_ms_dispatch(Message*)+0x554) [0x55cda5c0f664]
   21: (Monitor::ms_dispatch(Message*)+0x23) [0x55cda5c326f3]
   22: (DispatchQueue::entry()+0xf2b) [0x55cda60aedfb]
   23: (DispatchQueue::DispatchThread::entry()+0xd) [0x55cda5fa032d]
   24: (()+0x76fa) [0x7f6e4165a6fa]
   25: (clone()+0x6d) [0x7f6e3f916b5d]
   NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

  --- begin dump of recent events ---
  0> 2016-06-29 14:10:30.346791 7f6e3951d700 -1 ** Caught signal (Aborted) *
  in thread 7f6e3951d700 thread_name:ms_dispatch

  ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
   1: (()+0x5233be) [0x55cda5e8b3be]
   2: (()+0x113d0) [0x7f6e416643d0]
   3: (gsignal()+0x38) [0x7f6e3f845418]
   4: (abort()+0x16a) [0x7f6e3f84701a]
   5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55cda5fba18b]
   6: (()+0x560833) [0x55cda5ec8833]
   7: (CrushWrapper::move_bucket(CephContext*, int, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&)+0xda) [0x55cda5ec644a]
   8: (OSDMonitor::prepare_command_impl(std::shared_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > >&)+0x2cfe) [0x55cda5c8701e]
   9: (OSDMonitor::prepare_command(std::shared_ptr<MonOpRequest>)+0x2ff) [0x55cda5c9903f]
   10: (OSDMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x24b) [0x55cda5c9958b]
   11: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xb4f) [0x55cda5c4c0af]
   12: (PaxosService::C_RetryMessage::_finish(int)+0x58) [0x55cda5c4d698]
   13: (C_MonOp::finish(int)+0x82) [0x55cda5c15862]
   14: (Context::complete(int)+0x9) [0x55cda5c14949]
   15: (void finish_contexts<Context>(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0x1fb) [0x55cda5c1b25b]
   16: (Paxos::finish_round()+0x287) [0x55cda5c41b17]
   17: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xe19) [0x55cda5c42cf9]
   18: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x250) [0x55cda5c43520]
   19: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xa38) [0x55cda5c0ee68]
   20: (Monitor::_ms_dispatch(Message*)+0x554) [0x55cda5c0f664]
   21: (Monitor::ms_dispatch(Message*)+0x23) [0x55cda5c326f3]
   22: (DispatchQueue::entry()+0xf2b) [0x55cda60aedfb]
   23: (DispatchQueue::DispatchThread::entry()+0xd) [0x55cda5fa032d]
   24: (()+0x76fa) [0x7f6e4165a6fa]
   25: (clone()+0x6d) [0x7f6e3f916b5d]
   NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

  --- logging levels ---
  0/ 5 none
  0/ 1 lockdep
  0/ 1 context
  1/ 1 crush
  1/ 5 mds
  1/ 5 mds_balancer
  1/ 5 mds_locker
  1/ 5 mds_log
  1/ 5 mds_log_expire
  1/ 5 mds_migrator
  0/ 1 buffer
  0/ 1 timer
  0/ 1 filer
  0/ 1 striper
  0/ 1 objecter
  0/ 5 rados
  0/ 5 rbd
  0/ 5 rbd_mirror
  0/ 5 rbd_replay
  0/ 5 journaler
  0/ 5 objectcacher
  0/ 5 client
  0/ 5 osd
  0/ 5 optracker
  0/ 5 objclass
  1/ 3 filestore
  1/ 3 journal
  0/ 5 ms
  1/ 5 mon
  0/10 monc
  1/ 5 paxos
  0/ 5 tp
  1/ 5 auth
  1/ 5 crypto
  1/ 1 finisher
  1/ 5 heartbeatmap
  1/ 5 perfcounter
  1/ 5 rgw
  1/10 civetweb
  1/ 5 javaclient
  1/ 5 asok
  1/ 1 throttle
  0/ 0 refs
  1/ 5 xio
  1/ 5 compressor
  1/ 5 newstore
  1/ 5 bluestore
  1/ 5 bluefs
  1/ 3 bdev
  1/ 5 kstore
  4/ 5 rocksdb
  4/ 5 leveldb
  1/ 5 kinetic
  1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent 10000
  max_new 1000
  log_file /var/log/ceph/ceph-mon.pp5.log
  ```

  osd tree at the moment of crash:

  ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
  -17 1.00000 root fast2500
  -11 1.00000 host pp7
  0 1.00000 osd.0 up 1.00000 1.00000
  -5 9.12993 root ssd
  -4 4.79999 host pp11
  11 1.20000 osd.11 up 1.00000 1.00000
  3 1.20000 osd.3 up 1.00000 1.00000
  2 1.20000 osd.2 up 1.00000 1.00000
  1 1.20000 osd.1 up 1.00000 1.00000
  -8 0 host pp2
  -7 0 host pp3
  -12 0.25000 host pp4
  8 0.25000 osd.8 up 1.00000 1.00000
  -13 0.48000 host pp1
  4 0.48000 osd.4 up 1.00000 1.00000
  -2 0.09999 host c2
  9 0.09999 osd.9 up 0.79999 1.00000
  -1 0.09999 host c1
  6 0.09999 osd.6 up 1.00000 1.00000
  -3 0.09999 host c3
  10 0.09999 osd.10 up 1.00000 1.00000
  -6 0.70000 host c4
  12 0.70000 osd.12 up 0.79999 1.00000
  -9 0.45000 host c5
  13 0.45000 osd.13 up 1.00000 1.00000
  -10 0.45000 host c6
  14 0.45000 osd.14 up 1.00000 1.00000
  -14 0.45000 host c7
  15 0.45000 osd.15 up 1.00000 1.00000
  -15 0.79999 host c8
  16 0.79999 osd.16 up 1.00000 1.00000
  -16 0.45000 host c9
  17 0.45000 osd.17 up 1.00000 1.00000

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1597411/+subscriptions
  Thread Previous • Date Previous • • Thread Next