yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1670522] Re: TypeError booting instance with libvirt+xen in _create_pty_device with libvirt>=1.3.3

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Mon, 06 Mar 2017 23:39:54 -0000
Reply-to: Bug 1670522 <1670522@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
By the way we didn't catch this in xenproject CI because it's running an
older libvirt (1.3.1):

http://logs.openstack.xenproject.org/10/396210/5/check/dsvm-tempest-
xen/d4ed0ef/logs/dpkg-l.txt.gz

** Also affects: nova/ocata
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1670522

Title:
  TypeError booting instance with libvirt+xen in _create_pty_device with
  libvirt>=1.3.3

Status in OpenStack Compute (nova):
  Triaged
Status in OpenStack Compute (nova) ocata series:
  Confirmed

Bug description:
  Someone reported this in the nova IRC channel today using ocata with
  libvirt+xen:

  http://paste.openstack.org/raw/601670/

  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [req-937dedd1-35a1-46e4-8516-fc53c99a8f48 - - - - -] [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c] Instance failed to spawn
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c] Traceback (most recent call last):
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2125, in _build_resources
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]     yield resources
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]     block_device_info=block_device_info)
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2683, in spawn
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]     block_device_info=block_device_info)
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4871, in _get_guest_xml
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]     xml = conf.to_xml()
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py", line 77, in to_xml
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]     root = self.format_dom()
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py", line 2161, in format_dom
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]     self._format_devices(root)
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py", line 2119, in _format_devices
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]     devices.append(dev.format_dom())
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py", line 1636, in format_dom
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]     dev = super(LibvirtConfigGuestChar, self).format_dom()
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py", line 1622, in format_dom
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]     dev.append(self.log.format_dom())
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py", line 1665, in format_dom
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]     log.set("file", self.file)
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "src/lxml/lxml.etree.pyx", line 824, in lxml.etree._Element.set (src/lxml/lxml.etree.c:53073)
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "src/lxml/apihelpers.pxi", line 570, in lxml.etree._setAttributeValue (src/lxml/lxml.etree.c:23009)
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c]   File "src/lxml/apihelpers.pxi", line 1437, in lxml.etree._utf8 (src/lxml/lxml.etree.c:32414)
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c] TypeError: Argument must be bytes or unicode, got 'NoneType'
  2017-03-06 17:09:47.609 8530 ERROR nova.compute.manager [instance: a2ac1972-54c4-4a7b-ab34-04cce319806c] 

  The problem is when adding guest consoles to the domain xml during
  spawn, for libvirt+xen we go here:

  https://github.com/openstack/nova/blob/15.0.0/nova/virt/libvirt/driver.py#L4435

  And log_path=None in _create_pty_device:

  https://github.com/openstack/nova/blob/15.0.0/nova/virt/libvirt/driver.py#L4478

  CONF.serial_console.enabled is False and they have new enough
  libvirt/qemu for virtlogd support so they get here:

  https://github.com/openstack/nova/blob/15.0.0/nova/virt/libvirt/driver.py#L4510

  So we set log.file to None here:

  https://github.com/openstack/nova/blob/15.0.0/nova/virt/libvirt/driver.py#L4489

  Which blows up eventually because it's None.

  The workaround for now is to set [serial_console]enabled=True in
  nova.conf.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1670522/+subscriptions
References

[Bug 1670522] [NEW] TypeError booting instance with libvirt+xen in _create_pty_device with libvirt>=1.3.3
From: Matt Riedemann, 2017-03-06