← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #82906

[Bug 1880509] Re: Failed to attach volume - Connection to the hypervisor is broken on host

  Thread Previous • Date Previous • • Thread Next 
Marked invalid for kolla-ansible as it does not seem to involve us,
please change it if you find it can/has to be fixed in k-a.

** Changed in: kolla-ansible
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1880509

Title:
  Failed to attach volume - Connection to the hypervisor is broken on
  host

Status in kolla-ansible:
  Invalid
Status in OpenStack Compute (nova):
  New

Bug description:
  Using Rocky centos/source containers
  Docker version - 18.06.1-ce
  Hosts are Ubuntu 18.04.4 LTS

  Receiving the following defect on nova_compute which only occurs when attempting to attach an existing and unattached volume to a VM.
  -  Instances are created on the same host without issue and access to the virtual host is accessible via Virtual Machine Manager.  For testing purposes, I was able to attach the same volume via VMM.
  -  All compute, block, and network services are enabled and up.
  - 'nova hypervisor-list' shows the hypervisor up
  - The same volume and other volumes attach to VMs on a different hypervisor without issue.
  - The only difference between hypervisor hostnames that I see between the one that is failing and the one that is not is that the one that is failing is showing a FQDN.  Note below, it specifies only the hostname, not the FQDN.
  - From nova_compute, both the host name or the FQDN resolve correctly to the same IP and I can ping either.
  - virt_type is kvm
  - compute_driver is libvirt.LibvirtDriver
  - kernel on both hosts (failing and not failing to attach) are 4.15.0-101-generic

  Is this a settings issue, a permissions issue, a driver issue, or
  something else?

  From nova-compute.log:

  020-05-24 23:52:54.692 7 DEBUG oslo_concurrency.lockutils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lock "d51fddde-1b0c-4365-b1ec-ad508d2d0bed" released by "nova.compute.manager.do_reserve" :: held 0.154s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
  2020-05-24 23:52:55.469 7 DEBUG oslo_concurrency.lockutils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lock "d51fddde-1b0c-4365-b1ec-ad508d2d0bed" acquired by "nova.compute.manager.do_attach_volume" :: waited 0.000s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
  2020-05-24 23:52:55.470 7 INFO nova.compute.manager [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Attaching volume cadca51f-ff52-4b12-9777-4f86514e3eef to /dev/vdb
  2020-05-24 23:52:55.471 7 DEBUG nova.objects.instance [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] Lazy-loading 'flavor' on Instance uuid d51fddde-1b0c-4365-b1ec-ad508d2d0bed obj_load_attr /var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py:1111
  2020-05-24 23:52:55.738 7 DEBUG os_brick.utils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] ==> get_connector_properties: call u"{'execute': None, 'my_ip': '192.168.3.252', 'enforce_multipath': True, 'host': 'st1', 'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'multipath': False}" trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:146
  2020-05-24 23:52:56.428 7 DEBUG os_brick.initiator.linuxfc [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] No Fibre Channel support detected on system. get_fc_hbas /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157
  2020-05-24 23:52:56.429 7 DEBUG os_brick.initiator.linuxfc [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] No Fibre Channel support detected on system. get_fc_hbas /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157
  2020-05-24 23:52:56.430 7 DEBUG os_brick.utils [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] <== get_connector_properties: return (691ms) {'initiator': u'iqn.1994-05.com.redhat:cddc7297faec', 'ip': u'192.168.3.252', 'platform': u'x86_64', 'host': u'st1', 'do_local_attach': False, 'os_type': u'linux2', 'multipath': False} trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:170
  2020-05-24 23:52:56.432 7 DEBUG nova.virt.block_device [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Updating existing volume attachment record: 9a69f304-8468-47f6-8b59-5edd1e3dd75f _volume_attach /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py:535
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [req-94fdd34c-7799-4e1b-835e-78ee27236d71 240c5b223f324b5b84471039304fc530 96d55c41d2094f679c37078b034f552e - default default] [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Driver failed to attach volume cadca51f-ff52-4b12-9777-4f86514e3eef at /dev/vdb: HypervisorUnavailable: Connection to the hypervisor is broken on host: st1
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] Traceback (most recent call last):
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 563, in _volume_attach
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]     device_type=self['device_type'], encryption=encryption)
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1481, in attach_volume
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]     encryption=encryption)
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1264, in _connect_volume
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]     vol_driver.connect_volume(connection_info, instance)
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/fs.py", line 117, in connect_volume
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]     self._mount_options(connection_info))
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 409, in mount
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]     with __manager__.get_state() as mount_state:
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]     return self.gen.next()
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/volume/mount.py", line 89, in get_state
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]     raise exception.HypervisorUnavailable(host=CONF.host)
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed] HypervisorUnavailable: Connection to the hypervisor is broken on host: st1
  2020-05-24 23:53:09.078 7 ERROR nova.virt.block_device [instance: d51fddde-1b0c-4365-b1ec-ad508d2d0bed]

To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla-ansible/+bug/1880509/+subscriptions
  Thread Previous • Date Previous • • Thread Next