yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #88791
[Bug 1971663] [NEW] nova-compute process stuck without any actions
Public bug reported:
Description
===========
nova-compute process gets stuck after running for a while
Steps to reproduce
==================
Sorry, I haven't specified how to reproduce，but I know this problem has occurred at least three times in the arm environment, but not on x86.
Expected result
===============
nova-compute run normally
Actual result
=============
2022-04-28 15:07:15.150 5967 DEBUG oslo_concurrency.lockutils [-] Lock "ca12badc-42a2-43f2-8a28-cb2afa3a377c" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "53988d8e-75bc-40de-b9af-ce7509f94918" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "bc2b5ad1-a7a4-45fc-8d95-e986476c3129" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "f2068deb-c12f-4c31-998c-cf48af4440ff" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "48577b65-28e2-4858-abed-db00f90c0dc6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "5e6a68fc-b711-4115-bc77-179a224ed5c6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "4f47003a-c056-4aa6-9be2-e5bec0cb4a07" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "a1416265-68cb-4e06-a1b9-80c0bd138a09" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "96408c82-5951-46e6-b105-bcb21118ed99" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 17:11:03.055 5967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
timer()
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
cb(*args, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
waiter.switch()
greenlet.error: cannot switch to a different thread
Environment
===========
Python 3.6.8
eventlet 0.30.2
oslo.concurrency 4.4.0
oslo.messaging 12.7.1
nova-compute Stack list
======================
Stack for thread 281471352828416
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
time.sleep(seconds)
Stack for thread 281473394471424
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
time.sleep(seconds)
Stack for thread 281471310557696
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
time.sleep(seconds)
Stack for thread 281471838974464
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
time.sleep(seconds)
Stack for thread 281471847428608
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
time.sleep(seconds)
Stack for thread 281471855882752
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471864336896
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471872791040
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471881245184
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471889699328
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472375845376
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472384299520
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472392753664
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472401207808
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472409661952
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472418116096
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472426570240
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472979825152
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472988279296
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472996733440
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473005187584
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473013641728
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473022095872
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473030550016
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473111552512
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473120006656
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/nova/virt/libvirt/host.py", line 205, in _native_thread
libvirt.virEventRunDefaultImpl()
File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 442, in virEventRunDefaultImpl
ret = libvirtmod.virEventRunDefaultImpl()
Stack for thread 281473614563696
File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 42, in __thread_body
func(*args, **kwargs)
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 63, in wrap_bootstrap_inner
bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "<string>", line 167, in run
File "/usr/lib64/python3.6/code.py", line 233, in interact
more = self.push(line)
File "/usr/lib64/python3.6/code.py", line 259, in push
more = self.runsource(source, self.filename)
File "/usr/lib64/python3.6/code.py", line 75, in runsource
self.runcode(code)
File "/usr/lib64/python3.6/code.py", line 91, in runcode
exec(code, self.locals)
File "<console>", line 3, in <module>
========================================================
I tried to analyze why I got stuck, but it took me a long time to figure
it out, so looking for community help, thanks.
** Affects: nova
Importance: Undecided
Status: New
** Description changed:
Description
===========
nova-compute process gets stuck after running for a while
-
Steps to reproduce
==================
Sorry, I haven't specified how to reproduce，but I know this problem has occurred at least three times in the arm environment, but not on x86.
-
Expected result
===============
nova-compute run normally
-
Actual result
=============
2022-04-28 15:07:15.150 5967 DEBUG oslo_concurrency.lockutils [-] Lock "ca12badc-42a2-43f2-8a28-cb2afa3a377c" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "53988d8e-75bc-40de-b9af-ce7509f94918" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "bc2b5ad1-a7a4-45fc-8d95-e986476c3129" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "f2068deb-c12f-4c31-998c-cf48af4440ff" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "48577b65-28e2-4858-abed-db00f90c0dc6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "5e6a68fc-b711-4115-bc77-179a224ed5c6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "4f47003a-c056-4aa6-9be2-e5bec0cb4a07" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "a1416265-68cb-4e06-a1b9-80c0bd138a09" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "96408c82-5951-46e6-b105-bcb21118ed99" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 17:11:03.055 5967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
Traceback (most recent call last):
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
- timer()
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
- cb(*args, **kw)
- File "/usr/local/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
- waiter.switch()
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
+ timer()
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
+ cb(*args, **kw)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
+ waiter.switch()
greenlet.error: cannot switch to a different thread
-
Environment
===========
Python 3.6.8
eventlet 0.30.2
oslo.concurrency 4.4.0
oslo.messaging 12.7.1
-
nova-compute Stack list
======================
- i
+
Stack for thread 281471352828416
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
- self.wait(sleep_time)
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
- time.sleep(seconds)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
+ self.wait(sleep_time)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
+ time.sleep(seconds)
Stack for thread 281473394471424
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
- self.wait(sleep_time)
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
- time.sleep(seconds)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
+ self.wait(sleep_time)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
+ time.sleep(seconds)
Stack for thread 281471310557696
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
- self.wait(sleep_time)
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
- time.sleep(seconds)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
+ self.wait(sleep_time)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
+ time.sleep(seconds)
Stack for thread 281471838974464
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
- self.wait(sleep_time)
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
- time.sleep(seconds)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
+ self.wait(sleep_time)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
+ time.sleep(seconds)
Stack for thread 281471847428608
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
- self.wait(sleep_time)
- File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
- time.sleep(seconds)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
+ self.wait(sleep_time)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
+ time.sleep(seconds)
Stack for thread 281471855882752
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281471864336896
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281471872791040
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281471881245184
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281471889699328
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281472375845376
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281472384299520
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281472392753664
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281472401207808
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281472409661952
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281472418116096
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281472426570240
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281472979825152
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281472988279296
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281472996733440
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281473005187584
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281473013641728
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281473022095872
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281473030550016
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281473111552512
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
- msg = _reqq.get()
- File "/usr/lib64/python3.6/queue.py", line 164, in get
- self.not_empty.wait()
- File "/usr/lib64/python3.6/threading.py", line 295, in wait
- waiter.acquire()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
+ msg = _reqq.get()
+ File "/usr/lib64/python3.6/queue.py", line 164, in get
+ self.not_empty.wait()
+ File "/usr/lib64/python3.6/threading.py", line 295, in wait
+ waiter.acquire()
Stack for thread 281473120006656
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "/usr/lib64/python3.6/threading.py", line 864, in run
- self._target(*self._args, **self._kwargs)
- File "/usr/local/lib/python3.6/site-packages/nova/virt/libvirt/host.py", line 205, in _native_thread
- libvirt.virEventRunDefaultImpl()
- File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 442, in virEventRunDefaultImpl
- ret = libvirtmod.virEventRunDefaultImpl()
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "/usr/lib64/python3.6/threading.py", line 864, in run
+ self._target(*self._args, **self._kwargs)
+ File "/usr/local/lib/python3.6/site-packages/nova/virt/libvirt/host.py", line 205, in _native_thread
+ libvirt.virEventRunDefaultImpl()
+ File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 442, in virEventRunDefaultImpl
+ ret = libvirtmod.virEventRunDefaultImpl()
Stack for thread 281473614563696
- File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 42, in __thread_body
- func(*args, **kwargs)
- File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
- self._bootstrap_inner()
- File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 63, in wrap_bootstrap_inner
- bootstrap_inner()
- File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
- self.run()
- File "<string>", line 167, in run
- File "/usr/lib64/python3.6/code.py", line 233, in interact
- more = self.push(line)
- File "/usr/lib64/python3.6/code.py", line 259, in push
- more = self.runsource(source, self.filename)
- File "/usr/lib64/python3.6/code.py", line 75, in runsource
- self.runcode(code)
- File "/usr/lib64/python3.6/code.py", line 91, in runcode
- exec(code, self.locals)
- File "<console>", line 3, in <module>
-
+ File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 42, in __thread_body
+ func(*args, **kwargs)
+ File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
+ self._bootstrap_inner()
+ File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 63, in wrap_bootstrap_inner
+ bootstrap_inner()
+ File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
+ self.run()
+ File "<string>", line 167, in run
+ File "/usr/lib64/python3.6/code.py", line 233, in interact
+ more = self.push(line)
+ File "/usr/lib64/python3.6/code.py", line 259, in push
+ more = self.runsource(source, self.filename)
+ File "/usr/lib64/python3.6/code.py", line 75, in runsource
+ self.runcode(code)
+ File "/usr/lib64/python3.6/code.py", line 91, in runcode
+ exec(code, self.locals)
+ File "<console>", line 3, in <module>
========================================================
I tried to analyze why I got stuck, but it took me a long time to figure
it out, so looking for community help, thanks.
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1971663
Title:
nova-compute process stuck without any actions
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
nova-compute process gets stuck after running for a while
Steps to reproduce
==================
Sorry, I haven't specified how to reproduce，but I know this problem has occurred at least three times in the arm environment, but not on x86.
Expected result
===============
nova-compute run normally
Actual result
=============
2022-04-28 15:07:15.150 5967 DEBUG oslo_concurrency.lockutils [-] Lock "ca12badc-42a2-43f2-8a28-cb2afa3a377c" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "53988d8e-75bc-40de-b9af-ce7509f94918" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "bc2b5ad1-a7a4-45fc-8d95-e986476c3129" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "f2068deb-c12f-4c31-998c-cf48af4440ff" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "48577b65-28e2-4858-abed-db00f90c0dc6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "5e6a68fc-b711-4115-bc77-179a224ed5c6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "4f47003a-c056-4aa6-9be2-e5bec0cb4a07" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "a1416265-68cb-4e06-a1b9-80c0bd138a09" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "96408c82-5951-46e6-b105-bcb21118ed99" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
2022-04-28 17:11:03.055 5967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
timer()
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
cb(*args, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
waiter.switch()
greenlet.error: cannot switch to a different thread
Environment
===========
Python 3.6.8
eventlet 0.30.2
oslo.concurrency 4.4.0
oslo.messaging 12.7.1
nova-compute Stack list
======================
Stack for thread 281471352828416
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
time.sleep(seconds)
Stack for thread 281473394471424
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
time.sleep(seconds)
Stack for thread 281471310557696
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
time.sleep(seconds)
Stack for thread 281471838974464
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
time.sleep(seconds)
Stack for thread 281471847428608
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait
time.sleep(seconds)
Stack for thread 281471855882752
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471864336896
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471872791040
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471881245184
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281471889699328
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472375845376
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472384299520
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472392753664
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472401207808
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472409661952
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472418116096
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472426570240
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472979825152
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472988279296
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281472996733440
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473005187584
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473013641728
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473022095872
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473030550016
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473111552512
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker
msg = _reqq.get()
File "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire()
Stack for thread 281473120006656
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/nova/virt/libvirt/host.py", line 205, in _native_thread
libvirt.virEventRunDefaultImpl()
File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 442, in virEventRunDefaultImpl
ret = libvirtmod.virEventRunDefaultImpl()
Stack for thread 281473614563696
File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 42, in __thread_body
func(*args, **kwargs)
File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 63, in wrap_bootstrap_inner
bootstrap_inner()
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "<string>", line 167, in run
File "/usr/lib64/python3.6/code.py", line 233, in interact
more = self.push(line)
File "/usr/lib64/python3.6/code.py", line 259, in push
more = self.runsource(source, self.filename)
File "/usr/lib64/python3.6/code.py", line 75, in runsource
self.runcode(code)
File "/usr/lib64/python3.6/code.py", line 91, in runcode
exec(code, self.locals)
File "<console>", line 3, in <module>
========================================================
I tried to analyze why I got stuck, but it took me a long time to
figure it out, so looking for community help, thanks.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1971663/+subscriptions
Follow ups
