yahoo-eng-team team mailing list archive

[Jared Winborne <winborne.j.w@xxxxxxxxx>]

Public bug reported:

This issue has actually been caused by both Nova and libguestfs.

This has been tested on Rocky with libvirt.inject_key and
libvirt.inject_password enabled.

The error in nova-compute.log is the following.

2019-04-01 17:02:50.325 2925194 WARNING nova.virt.disk.api [req-fcaf4276-3be9-4218-b425-a8bb2ffc6739 c9765e181f304bf6b7a6077b860d2eac f4a2a6a655cd4fde9d878ae750078e11 - default default] Ignoring error injecting data into image <RBDImage:
{'name': u'7dbca304-535d-4872-939a-29f5e810f948_disk', 'format': 'raw', 'servers': ['fd00::cefc:1:6789', 'fd00::cefc:2:6789', 'fd00::cefc:3:6789'], 'user': None, 'password': None, 'pool': 'vms'}> (Error mounting <RBDImage:{'name': u'7dbc
a304-535d-4872-939a-29f5e810f948_disk', 'format': 'raw', 'servers': ['fd00::cefc:1:6789', 'fd00::cefc:2:6789', 'fd00::cefc:3:6789'], 'user': None, 'password': None, 'pool': 'vms'}> with libguestfs (guestfs_launch failed.
This usually means the libguestfs appliance failed to start or crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.)): NovaException: Error mounting <RBDImage:{'name': u'7dbca304-535d-4872-939a-29f5e810f948_disk', 'format': 'raw', 'servers': ['fd00::cefc:1:6789', 'fd00::cefc:2:6789', 'fd00::
cefc:3:6789'], 'user': None, 'password': None, 'pool': 'vms'}> with libguestfs (guestfs_launch failed.
This usually means the libguestfs appliance failed to start or crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.)

This is actually being caused by the IPv6 addresses not having brackets,
as it should be '[fd00::cefc:1]:6789'

Even when formatted correctly, libguestfs wouldn't accept these but this was actually fixed earlier today. 
https://github.com/libguestfs/libguestfs/commit/e047cc4da8c69f20b97965268b2b951ebbed0232 

DNS with AAAA records would work, but it gets the addresses directly
from ceph and has to be addresses.

I have been able to track this down for the most part. The addresses are
retrieved from ceph in the get_mon_addrs method and the brackets are
removed :
https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/storage/rbd_utils.py#n163

This is used at here to make the dict from the error message in the get_model method:
https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/imagebackend.py#n956

The 'servers' variable there is the one used. As best as I've been able
to tell, it is only used for libguestfs.

It looks like the loop here would need to be expanded to check for IPv6 and add brackets accordingly.
https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/imagebackend.py#n964

I'm unsure of the best way to handle this, but since Ceph should only return IPv4 or IPv6 addresses, something like this could be done right before 964 to modify the list that is used to make 'servers'
hosts = [ '[%s]' % k if ':' in k else k for k in hosts ]

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1822689

Title:
  libguestfs data injection fails with Ceph and IPv6

Status in OpenStack Compute (nova):
  New

Bug description:
  This issue has actually been caused by both Nova and libguestfs.

  This has been tested on Rocky with libvirt.inject_key and
  libvirt.inject_password enabled.

  The error in nova-compute.log is the following.

  2019-04-01 17:02:50.325 2925194 WARNING nova.virt.disk.api [req-fcaf4276-3be9-4218-b425-a8bb2ffc6739 c9765e181f304bf6b7a6077b860d2eac f4a2a6a655cd4fde9d878ae750078e11 - default default] Ignoring error injecting data into image <RBDImage:
  {'name': u'7dbca304-535d-4872-939a-29f5e810f948_disk', 'format': 'raw', 'servers': ['fd00::cefc:1:6789', 'fd00::cefc:2:6789', 'fd00::cefc:3:6789'], 'user': None, 'password': None, 'pool': 'vms'}> (Error mounting <RBDImage:{'name': u'7dbc
  a304-535d-4872-939a-29f5e810f948_disk', 'format': 'raw', 'servers': ['fd00::cefc:1:6789', 'fd00::cefc:2:6789', 'fd00::cefc:3:6789'], 'user': None, 'password': None, 'pool': 'vms'}> with libguestfs (guestfs_launch failed.
  This usually means the libguestfs appliance failed to start or crashed.
  Do:
    export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
  and run the command again.  For further information, read:
    http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
  You can also run 'libguestfs-test-tool' and post the *complete* output
  into a bug report or message to the libguestfs mailing list.)): NovaException: Error mounting <RBDImage:{'name': u'7dbca304-535d-4872-939a-29f5e810f948_disk', 'format': 'raw', 'servers': ['fd00::cefc:1:6789', 'fd00::cefc:2:6789', 'fd00::
  cefc:3:6789'], 'user': None, 'password': None, 'pool': 'vms'}> with libguestfs (guestfs_launch failed.
  This usually means the libguestfs appliance failed to start or crashed.
  Do:
    export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
  and run the command again.  For further information, read:
    http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
  You can also run 'libguestfs-test-tool' and post the *complete* output
  into a bug report or message to the libguestfs mailing list.)

  This is actually being caused by the IPv6 addresses not having
  brackets, as it should be '[fd00::cefc:1]:6789'

  Even when formatted correctly, libguestfs wouldn't accept these but this was actually fixed earlier today. 
  https://github.com/libguestfs/libguestfs/commit/e047cc4da8c69f20b97965268b2b951ebbed0232 

  DNS with AAAA records would work, but it gets the addresses directly
  from ceph and has to be addresses.

  I have been able to track this down for the most part. The addresses
  are retrieved from ceph in the get_mon_addrs method and the brackets
  are removed :
  https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/storage/rbd_utils.py#n163

  This is used at here to make the dict from the error message in the get_model method:
  https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/imagebackend.py#n956

  The 'servers' variable there is the one used. As best as I've been
  able to tell, it is only used for libguestfs.

  It looks like the loop here would need to be expanded to check for IPv6 and add brackets accordingly.
  https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/imagebackend.py#n964

  I'm unsure of the best way to handle this, but since Ceph should only return IPv4 or IPv6 addresses, something like this could be done right before 964 to modify the list that is used to make 'servers'
  hosts = [ '[%s]' % k if ':' in k else k for k in hosts ]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1822689/+subscriptions