yahoo-eng-team team mailing list archive

[OpenStack Infra <1842081@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/679473
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3cef7edbc2dd72a25e46ca1962731ee3153f1fd7
Submitter: Zuul
Branch:    master

commit 3cef7edbc2dd72a25e46ca1962731ee3153f1fd7
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Fri Aug 30 12:11:59 2019 -0400

    Handle VirtDriverNotReady in _cleanup_running_deleted_instances
    
    Change I6e086e0a2bffe12fd78373b3bc6215221dabc77e made the
    _cleanup_running_deleted_instances task run immediately on
    start of the nova-compute service. For some compute drivers, like
    ironic, the "hypervisor" might not yet be ready to list instances
    which results in a traceback exception in the logs.
    
    This simply handles the VirtDriverNotReady error and returns to
    avoid the traceback.
    
    Change-Id: I6220a59988502a9839645efa101d555c19d4b650
    Closes-Bug: #1842081


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1842081

Title:
  Error during ComputeManager._cleanup_running_deleted_instances:
  VirtDriverNotReady: Virt driver is not ready. (ironic)

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Seeing this on start of nova-compute with ironic when ironic-api isn't
  yet available:

  Aug 24 01:06:39.710754 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR nova.virt.ironic.driver [None req-
  9542c6c8-a038-45f5-bd18-e18f83c17755 None None] An unknown error has
  occurred when trying to get the list of nodes from the Ironic
  inventory. Error: StrictVersion instance has no attribute 'version'

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task [None req-
  9542c6c8-a038-45f5-bd18-e18f83c17755 None None] Error during
  ComputeManager._cleanup_running_deleted_instances: VirtDriverNotReady:
  Virt driver is not ready.

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task Traceback (most recent
  call last):

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/usr/local/lib/python2.7/dist-
  packages/oslo_service/periodic_task.py", line 222, in
  run_periodic_tasks

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task     task(self,
  context)

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/opt/stack/nova/nova/compute/manager.py", line 8369, in
  _cleanup_running_deleted_instances

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task     for instance in
  self._running_deleted_instances(context):

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/opt/stack/nova/nova/compute/manager.py", line 8423, in
  _running_deleted_instances

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task     instances =
  self._get_instances_on_driver(context, filters)

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/opt/stack/nova/nova/compute/manager.py", line 634, in
  _get_instances_on_driver

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task     driver_uuids =
  self.driver.list_instance_uuids()

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/opt/stack/nova/nova/virt/ironic/driver.py", line 685, in
  list_instance_uuids

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task
  fields=['instance_uuid'], limit=0)

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task   File
  "/opt/stack/nova/nova/virt/ironic/driver.py", line 656, in
  _get_node_list

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task     raise
  exception.VirtDriverNotReady()

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task VirtDriverNotReady:
  Virt driver is not ready.

  Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
  compute[7945]: ERROR oslo_service.periodic_task

  Looks like this is due to https://review.opendev.org/#/c/657132/ in
  Train where the _cleanup_running_deleted_instances periodic task runs
  immediately on startup of the nova-compute service which could be
  before the hypervisor (in this case ironic) is ready.

  This doesn't really break anything, but it's an ugly traceback in the
  logs that could be avoided. We should handle the VirtDriverNotReady
  error and return from the periodic.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1842081/+subscriptions