yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1915364] [NEW] fail to launch instances

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Martin <1915364@xxxxxxxxxxxxxxxxxx>
Date: Thu, 11 Feb 2021 03:39:53 -0000
Reply-to: Bug 1915364 <1915364@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I'm testing openstack on my home lab (3x Dell T410 loaded).  I've been
using lately openstack-ansible to deploy.  Besides some issues here and
there, the overall process succeed.  I'm presently hitting an issue that
I can't seem to find the possible cause.

Exhausted all hosts available for retrying ... I think this is a general
response always giving even if the issue is with the network, storage,
resource, etc.

FYI, I'm using ceph as a backend and it works good.
I'm able to create volumes, images, volumes-from-images, all from openstack command line and horizon.
I have networks which I can ping from outside openstack. 
All openstack services shows ok.  From the logs, I seem to have some issues with gnocchi, I don't know to what extend this coud prevents me from spawning instances.

gnocchi.incoming.SackDetectionError: [Errno 2] No such file or
directory: '/var/lib/gnocchi/tmp/gnocchi-config'

I have full logs, but I can't seem to find the culprit.

In the compute-node, I have this..pretty cryptic...:

ERROR nova.compute.manager [req-66f25425-c735-44dd-aa69-d8e1c904f17e 0540dc5ba00641c1b86a269b99d4b323 ddc8fdea71a943a7b36c4ae66b2f44ed - default default] [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95] Failed to build and run instance: libvirt.libvirtError: Failed to activate service 'org.freedesktop.machine1': timed out (service_start_timeout=25000ms)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95] Traceback (most recent call last):#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/compute/manager.py", line 2384, in _build_and_run_instance#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     self.driver.spawn(context, instance, image_meta,#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 3682, in spawn#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     self._create_guest_with_network(#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 6683, in _create_guest_with_network#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     self._cleanup_failed_start(#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     self.force_reraise()#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     six.reraise(self.type_, self.value, self.tb)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py", line 703, in reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     raise value#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 6652, in _create_guest_with_network#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     guest = self._create_guest(#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 6585, in _create_guest#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     guest.launch(pause=pause)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 158, in launch#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     LOG.exception('Error launching a defined domain with XML: %s',#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     self.force_reraise()#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     six.reraise(self.type_, self.value, self.tb)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py", line 703, in reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     raise value#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 155, in launch#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     return self._domain.createWithFlags(flags)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py", line 190, in doit#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     result = proxy_call(self._autowrap, f, *args, **kwargs)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py", line 148, in proxy_call#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     rv = execute(f, *args, **kwargs)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     six.reraise(c, e, tb)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py", line 703, in reraise#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     raise value#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     rv = meth(*args, **kwargs)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)#0122021-02-09 09:41:39.705 408599
ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95] libvirt.libvirtError: Failed to activate service 'org.freedesktop.machine1': timed out (service_start_timeout=25000ms)#0122021-02-09 09:41:39.705 408599

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1915364

Title:
  fail to launch instances

Status in OpenStack Compute (nova):
  New

Bug description:
  I'm testing openstack on my home lab (3x Dell T410 loaded).  I've been
  using lately openstack-ansible to deploy.  Besides some issues here
  and there, the overall process succeed.  I'm presently hitting an
  issue that I can't seem to find the possible cause.

  Exhausted all hosts available for retrying ... I think this is a
  general response always giving even if the issue is with the network,
  storage, resource, etc.

  FYI, I'm using ceph as a backend and it works good.
  I'm able to create volumes, images, volumes-from-images, all from openstack command line and horizon.
  I have networks which I can ping from outside openstack. 
  All openstack services shows ok.  From the logs, I seem to have some issues with gnocchi, I don't know to what extend this coud prevents me from spawning instances.

  gnocchi.incoming.SackDetectionError: [Errno 2] No such file or
  directory: '/var/lib/gnocchi/tmp/gnocchi-config'

  I have full logs, but I can't seem to find the culprit.

  In the compute-node, I have this..pretty cryptic...:

  ERROR nova.compute.manager [req-66f25425-c735-44dd-aa69-d8e1c904f17e 0540dc5ba00641c1b86a269b99d4b323 ddc8fdea71a943a7b36c4ae66b2f44ed - default default] [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95] Failed to build and run instance: libvirt.libvirtError: Failed to activate service 'org.freedesktop.machine1': timed out (service_start_timeout=25000ms)#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95] Traceback (most recent call last):#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/compute/manager.py", line 2384, in _build_and_run_instance#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     self.driver.spawn(context, instance, image_meta,#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 3682, in spawn#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     self._create_guest_with_network(#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 6683, in _create_guest_with_network#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     self._cleanup_failed_start(#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     self.force_reraise()#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     six.reraise(self.type_, self.value, self.tb)#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py", line 703, in reraise#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     raise value#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 6652, in _create_guest_with_network#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     guest = self._create_guest(#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 6585, in _create_guest#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     guest.launch(pause=pause)#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 158, in launch#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     LOG.exception('Error launching a defined domain with XML: %s',#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     self.force_reraise()#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     six.reraise(self.type_, self.value, self.tb)#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py", line 703, in reraise#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     raise value#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 155, in launch#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     return self._domain.createWithFlags(flags)#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py", line 190, in doit#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     result = proxy_call(self._autowrap, f, *args, **kwargs)#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py", line 148, in proxy_call#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     rv = execute(f, *args, **kwargs)#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     six.reraise(c, e, tb)#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/six.py", line 703, in reraise#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     raise value#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     rv = meth(*args, **kwargs)#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]   File "/openstack/venvs/nova-22.0.0.0rc2.dev1/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)#0122021-02-09 09:41:39.705 408599
  ERROR nova.compute.manager [instance: b721cf9a-e3f3-4b89-8750-1b9f1e273f95] libvirt.libvirtError: Failed to activate service 'org.freedesktop.machine1': timed out (service_start_timeout=25000ms)#0122021-02-09 09:41:39.705 408599

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1915364/+subscriptions
Follow ups

[Bug 1915364] Re: fail to launch instances
From: Balazs Gibizer, 2021-02-11