yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1845759] [NEW] Openstack nova-compute service gives cant start new thread with oslo messaging

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ankit Sambhare <1845759@xxxxxxxxxxxxxxxxxx>
Date: Sat, 28 Sep 2019 04:15:31 -0000
Reply-to: Bug 1845759 <1845759@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I already have a ocata openstack version installed on ubuntu OS and some
of my compute nodes are giving "cant start new thread erros with olso
messaging messaging" and my nova-compute services are killed but i cant
find any info in syslog for the same.

ERROR logs from Compute node :

2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service [-] Error starting thread.
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service Traceback (most recent call last):
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     service.start()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     self.manager.init_host()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1142, in init_host
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     self.driver.init_host(host=self.host)
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 491, in init_host
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     self._do_quality_warnings()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 468, in _do_quality_warnings
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     caps = self._host.get_capabilities()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 634, in get_capabilities
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     xmlstr = self.get_connection().getCapabilities()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 452, in get_connection
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     conn = self._get_connection()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 435, in _get_connection
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     {'msg': ex})
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     self.force_reraise()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     six.reraise(self.type_, self.value, self.tb)
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 424, in _get_connection
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     self._wrapped_conn = self._get_new_connection()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 369, in _get_new_connection
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     wrapped_conn = self._connect(self._uri, self._read_only)
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 225, in _connect
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     libvirt.openAuth, uri, auth, flags)
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     rv = execute(f, *args, **kwargs)
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 106, in execute
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     setup()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 287, in setup
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     t.start()
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/threading.py", line 736, in start
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     _start_new_thread(self.__bootstrap, ())
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service error: can't start new thread
2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service [-] Error starting thread.
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service Traceback (most recent call last):
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     service.start()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     self.manager.init_host()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1142, in init_host
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     self.driver.init_host(host=self.host)
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 491, in init_host
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     self._do_quality_warnings()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 468, in _do_quality_warnings
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     caps = self._host.get_capabilities()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 634, in get_capabilities
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     xmlstr = self.get_connection().getCapabilities()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 452, in get_connection
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     conn = self._get_connection()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 435, in _get_connection
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     {'msg': ex})
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     self.force_reraise()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     six.reraise(self.type_, self.value, self.tb)
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 424, in _get_connection
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     self._wrapped_conn = self._get_new_connection()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 369, in _get_new_connection
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     wrapped_conn = self._connect(self._uri, self._read_only)
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 225, in _connect
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     libvirt.openAuth, uri, auth, flags)
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     rv = execute(f, *args, **kwargs)
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 106, in execute
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     setup()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 287, in setup
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     t.start()
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/threading.py", line 736, in start
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     _start_new_thread(self.__bootstrap, ())
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service error: can't start new thread
2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "openstack nova logs and configs file"
   https://bugs.launchpad.net/bugs/1845759/+attachment/5292085/+files/sosreport-cmp120-20190928094143.tar.xz

** Summary changed:

- Openstack nova-compute service gives cant start ew thread with oslo messaging 
+ Openstack nova-compute service gives cant start new thread with oslo messaging

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1845759

Title:
  Openstack nova-compute service gives cant start new thread with oslo
  messaging

Status in OpenStack Compute (nova):
  New

Bug description:
  I already have a ocata openstack version installed on ubuntu OS and
  some of my compute nodes are giving "cant start new thread erros with
  olso messaging messaging" and my nova-compute services are killed but
  i cant find any info in syslog for the same.

  ERROR logs from Compute node :

  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service [-] Error starting thread.
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service Traceback (most recent call last):
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     service.start()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     self.manager.init_host()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1142, in init_host
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     self.driver.init_host(host=self.host)
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 491, in init_host
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     self._do_quality_warnings()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 468, in _do_quality_warnings
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     caps = self._host.get_capabilities()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 634, in get_capabilities
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     xmlstr = self.get_connection().getCapabilities()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 452, in get_connection
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     conn = self._get_connection()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 435, in _get_connection
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     {'msg': ex})
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     self.force_reraise()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     six.reraise(self.type_, self.value, self.tb)
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 424, in _get_connection
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     self._wrapped_conn = self._get_new_connection()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 369, in _get_new_connection
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     wrapped_conn = self._connect(self._uri, self._read_only)
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 225, in _connect
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     libvirt.openAuth, uri, auth, flags)
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     rv = execute(f, *args, **kwargs)
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 106, in execute
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     setup()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 287, in setup
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     t.start()
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service   File "/usr/lib/python2.7/threading.py", line 736, in start
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service     _start_new_thread(self.__bootstrap, ())
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service error: can't start new thread
  2019-09-27 20:43:20,010.010 3925 ERROR oslo_service.service
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service [-] Error starting thread.
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service Traceback (most recent call last):
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     service.start()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     self.manager.init_host()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1142, in init_host
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     self.driver.init_host(host=self.host)
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 491, in init_host
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     self._do_quality_warnings()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 468, in _do_quality_warnings
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     caps = self._host.get_capabilities()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 634, in get_capabilities
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     xmlstr = self.get_connection().getCapabilities()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 452, in get_connection
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     conn = self._get_connection()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 435, in _get_connection
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     {'msg': ex})
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     self.force_reraise()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     six.reraise(self.type_, self.value, self.tb)
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 424, in _get_connection
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     self._wrapped_conn = self._get_new_connection()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 369, in _get_new_connection
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     wrapped_conn = self._connect(self._uri, self._read_only)
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 225, in _connect
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     libvirt.openAuth, uri, auth, flags)
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     rv = execute(f, *args, **kwargs)
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 106, in execute
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     setup()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 287, in setup
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     t.start()
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service   File "/usr/lib/python2.7/threading.py", line 736, in start
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service     _start_new_thread(self.__bootstrap, ())
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service error: can't start new thread
  2019-09-28 00:53:27,722.722 8107 ERROR oslo_service.service

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1845759/+subscriptions
Follow ups

[Bug 1845759] Re: Openstack nova-compute service gives cant start new thread with oslo messaging
From: Launchpad Bug Tracker, 2020-06-21