yahoo-eng-team team mailing list archive

[Lee Yarwood <lyarwood@xxxxxxxxxx>]

Public bug reported:

Description
===========

I8af2ad741ca08c3d88efb9aa817c4d1470491a23 has started to correctly fence
the subnode during evacuation testing. However it missed that we deploy
c-vol and g-api on these nodes. As a result during BFV evacuation
testing we will fail if the volume has been created on the subnode
c-vol.

https://zuul.opendev.org/t/openstack/build/c78d3ab4e6a748b4a53c6ff6dc273106/log/logs/screen-n-cpu.txt#7060

Mar 19 19:43:26.844295 ubuntu-bionic-rax-ord-0015339373 nova-
compute[9838]: ERROR nova.compute.manager [None req-
512a96c8-8b32-49c7-8d29-7ff300ed4482 demo admin] [instance:
702ff125-d947-4a28-853b-82dcd58b990e] Setting instance vm_state to
ERROR: ClientException: The server has either erred or is incapable of
performing the requested operation. (HTTP 500)

https://zuul.opendev.org/t/openstack/build/c78d3ab4e6a748b4a53c6ff6dc273106/log/logs/screen-c-api.txt#1936

Mar 19 19:43:26.262818 ubuntu-bionic-rax-ord-0015339373 devstack@c-api.service[27200]: ERROR cinder.api.middleware.fault [req-512a96c8-8b32-49c7-8d29-7ff300ed4482 req-826f7c01-3c02-4d9e-9046-8a15d7fa9b61 demo admin] Caught error: <class 'oslo_messaging.exceptions.MessagingTimeout'> Timed out waiting for a reply to message ID 23fabce9b79441198fbe4fe71c0ac7ab: MessagingTimeout: Timed out waiting for a reply to message ID 23fabce9b79441198fbe4fe71c0ac7ab
Mar 19 19:43:26.262818 ubuntu-bionic-rax-ord-0015339373 devstack@c-api.service[27200]: ERROR 

Ultimately we shouldn't run these services on the computes but for now
we should limit the services we stop on the subnode to n-cpu and q-agt.

Steps to reproduce
==================
Run nova-live-migration, if volumes are created on the subnode evacuation testing will fail.

Expected result
===============
nova-live-migration passes.

Actual result
=============
nova-live-migration fails.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   Master or stabe/train with I8af2ad741ca08c3d88efb9aa817c4d1470491a23
applied.

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   N/A

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

** Affects: nova
     Importance: High
     Assignee: Lee Yarwood (lyarwood)
         Status: In Progress

** Affects: nova/stein
     Importance: Undecided
         Status: New

** Affects: nova/train
     Importance: Undecided
         Status: New

** Affects: nova/ussuri
     Importance: High
     Assignee: Lee Yarwood (lyarwood)
         Status: In Progress


** Tags: evacuate live-migration volumes

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1868234

Title:
  nova-live-migration evacuation fails if volumes created on subnode
  c-vol backend

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) stein series:
  New
Status in OpenStack Compute (nova) train series:
  New
Status in OpenStack Compute (nova) ussuri series:
  In Progress

Bug description:
  Description
  ===========

  I8af2ad741ca08c3d88efb9aa817c4d1470491a23 has started to correctly
  fence the subnode during evacuation testing. However it missed that we
  deploy c-vol and g-api on these nodes. As a result during BFV
  evacuation testing we will fail if the volume has been created on the
  subnode c-vol.

  https://zuul.opendev.org/t/openstack/build/c78d3ab4e6a748b4a53c6ff6dc273106/log/logs/screen-n-cpu.txt#7060

  Mar 19 19:43:26.844295 ubuntu-bionic-rax-ord-0015339373 nova-
  compute[9838]: ERROR nova.compute.manager [None req-
  512a96c8-8b32-49c7-8d29-7ff300ed4482 demo admin] [instance:
  702ff125-d947-4a28-853b-82dcd58b990e] Setting instance vm_state to
  ERROR: ClientException: The server has either erred or is incapable of
  performing the requested operation. (HTTP 500)

  https://zuul.opendev.org/t/openstack/build/c78d3ab4e6a748b4a53c6ff6dc273106/log/logs/screen-c-api.txt#1936

  Mar 19 19:43:26.262818 ubuntu-bionic-rax-ord-0015339373 devstack@c-api.service[27200]: ERROR cinder.api.middleware.fault [req-512a96c8-8b32-49c7-8d29-7ff300ed4482 req-826f7c01-3c02-4d9e-9046-8a15d7fa9b61 demo admin] Caught error: <class 'oslo_messaging.exceptions.MessagingTimeout'> Timed out waiting for a reply to message ID 23fabce9b79441198fbe4fe71c0ac7ab: MessagingTimeout: Timed out waiting for a reply to message ID 23fabce9b79441198fbe4fe71c0ac7ab
  Mar 19 19:43:26.262818 ubuntu-bionic-rax-ord-0015339373 devstack@c-api.service[27200]: ERROR 

  Ultimately we shouldn't run these services on the computes but for now
  we should limit the services we stop on the subnode to n-cpu and
  q-agt.

  Steps to reproduce
  ==================
  Run nova-live-migration, if volumes are created on the subnode evacuation testing will fail.

  Expected result
  ===============
  nova-live-migration passes.

  Actual result
  =============
  nova-live-migration fails.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     Master or stabe/train with
  I8af2ad741ca08c3d88efb9aa817c4d1470491a23 applied.

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libvirt + KVM

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     N/A

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1868234/+subscriptions