yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1842081] [NEW] Error during ComputeManager._cleanup_running_deleted_instances: VirtDriverNotReady: Virt driver is not ready. (ironic)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Fri, 30 Aug 2019 14:21:40 -0000
Reply-to: Bug 1842081 <1842081@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seeing this on start of nova-compute with ironic when ironic-api isn't
yet available:

Aug 24 01:06:39.710754 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR nova.virt.ironic.driver [None req-
9542c6c8-a038-45f5-bd18-e18f83c17755 None None] An unknown error has
occurred when trying to get the list of nodes from the Ironic inventory.
Error: StrictVersion instance has no attribute 'version'

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task [None req-
9542c6c8-a038-45f5-bd18-e18f83c17755 None None] Error during
ComputeManager._cleanup_running_deleted_instances: VirtDriverNotReady:
Virt driver is not ready.

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task Traceback (most recent
call last):

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task   File
"/usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py",
line 222, in run_periodic_tasks

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task     task(self, context)

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task   File
"/opt/stack/nova/nova/compute/manager.py", line 8369, in
_cleanup_running_deleted_instances

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task     for instance in
self._running_deleted_instances(context):

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task   File
"/opt/stack/nova/nova/compute/manager.py", line 8423, in
_running_deleted_instances

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task     instances =
self._get_instances_on_driver(context, filters)

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task   File
"/opt/stack/nova/nova/compute/manager.py", line 634, in
_get_instances_on_driver

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task     driver_uuids =
self.driver.list_instance_uuids()

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task   File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 685, in
list_instance_uuids

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task
fields=['instance_uuid'], limit=0)

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task   File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 656, in
_get_node_list

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task     raise
exception.VirtDriverNotReady()

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task VirtDriverNotReady: Virt
driver is not ready.

Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task

Looks like this is due to https://review.opendev.org/#/c/657132/ in
Train where the _cleanup_running_deleted_instances periodic task runs
immediately on startup of the nova-compute service which could be before
the hypervisor (in this case ironic) is ready.

This doesn't really break anything, but it's an ugly traceback in the
logs that could be avoided. We should handle the VirtDriverNotReady
error and return from the periodic.

** Affects: nova
     Importance: Low
     Assignee: Matt Riedemann (mriedem)
         Status: Triaged


** Tags: compute ironic

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1842081

Title:
  Error during ComputeManager._cleanup_running_deleted_instances:
  VirtDriverNotReady: Virt driver is not ready. (ironic)

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  Seeing this on start of nova-compute with ironic when ironic-api isn't
  yet available:

  Aug 24 01:06:39.710754 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR nova.virt.ironic.driver [None req-
  9542c6c8-a038-45f5-bd18-e18f83c17755 None None] An unknown error has
  occurred when trying to get the list of nodes from the Ironic
  inventory. Error: StrictVersion instance has no attribute 'version'

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task [None req-
  9542c6c8-a038-45f5-bd18-e18f83c17755 None None] Error during
  ComputeManager._cleanup_running_deleted_instances: VirtDriverNotReady:
  Virt driver is not ready.

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task Traceback (most recent
  call last):

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/usr/local/lib/python2.7/dist-
  packages/oslo_service/periodic_task.py", line 222, in
  run_periodic_tasks

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task     task(self,
  context)

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/opt/stack/nova/nova/compute/manager.py", line 8369, in
  _cleanup_running_deleted_instances

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task     for instance in
  self._running_deleted_instances(context):

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/opt/stack/nova/nova/compute/manager.py", line 8423, in
  _running_deleted_instances

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task     instances =
  self._get_instances_on_driver(context, filters)

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/opt/stack/nova/nova/compute/manager.py", line 634, in
  _get_instances_on_driver

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task     driver_uuids =
  self.driver.list_instance_uuids()

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/opt/stack/nova/nova/virt/ironic/driver.py", line 685, in
  list_instance_uuids

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task
  fields=['instance_uuid'], limit=0)

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/opt/stack/nova/nova/virt/ironic/driver.py", line 656, in
  _get_node_list

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task     raise
  exception.VirtDriverNotReady()

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task VirtDriverNotReady:
  Virt driver is not ready.

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task

  Looks like this is due to https://review.opendev.org/#/c/657132/ in
  Train where the _cleanup_running_deleted_instances periodic task runs
  immediately on startup of the nova-compute service which could be
  before the hypervisor (in this case ironic) is ready.

  This doesn't really break anything, but it's an ugly traceback in the
  logs that could be avoided. We should handle the VirtDriverNotReady
  error and return from the periodic.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1842081/+subscriptions
Follow ups

[Bug 1842081] Re: Error during ComputeManager._cleanup_running_deleted_instances: VirtDriverNotReady: Virt driver is not ready. (ironic)
From: OpenStack Infra, 2019-09-06