yahoo-eng-team team mailing list archive

[OpenStack Infra <1822689@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/649405
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5d29dccaf91c697ffc12a7289ce16cd83e60b07f
Submitter: Zuul
Branch:    master

commit 5d29dccaf91c697ffc12a7289ce16cd83e60b07f
Author: Jared Winborne <jwinborne@xxxxxxxxxxxxx>
Date:   Tue Apr 2 12:22:37 2019 -0500

    Leave brackets on Ceph IP addresses for libguestfs
    
    Brackets are removed from all IPv6 addresses when getting Ceph monitor
    addresses. This is correct for libvirt's XML files but not for
    libguestfs.
    
    This adds a 'strip_brackets' argument for get_mon_addrs method. It is
    default 'True' to not interfere anywhere this is used with no arguments.
    
    Uses strip_brackets=False when getting the hosts and ports for RBDImage,
    which is used for libguestfs.
    
    Change-Id: I378998a73046b9b9a33e30d0e06c4721d712da65
    Closes-Bug: 1822689


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1822689

Title:
  libguestfs data injection fails with Ceph and IPv6

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  This issue has actually been caused by both Nova and libguestfs.

  This has been tested on Rocky with libvirt.inject_key and
  libvirt.inject_password enabled.

  The error in nova-compute.log is the following.

  2019-04-01 17:02:50.325 2925194 WARNING nova.virt.disk.api [req-fcaf4276-3be9-4218-b425-a8bb2ffc6739 c9765e181f304bf6b7a6077b860d2eac f4a2a6a655cd4fde9d878ae750078e11 - default default] Ignoring error injecting data into image <RBDImage:
  {'name': u'7dbca304-535d-4872-939a-29f5e810f948_disk', 'format': 'raw', 'servers': ['fd00::cefc:1:6789', 'fd00::cefc:2:6789', 'fd00::cefc:3:6789'], 'user': None, 'password': None, 'pool': 'vms'}> (Error mounting <RBDImage:{'name': u'7dbc
  a304-535d-4872-939a-29f5e810f948_disk', 'format': 'raw', 'servers': ['fd00::cefc:1:6789', 'fd00::cefc:2:6789', 'fd00::cefc:3:6789'], 'user': None, 'password': None, 'pool': 'vms'}> with libguestfs (guestfs_launch failed.
  This usually means the libguestfs appliance failed to start or crashed.
  Do:
    export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
  and run the command again.  For further information, read:
    http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
  You can also run 'libguestfs-test-tool' and post the *complete* output
  into a bug report or message to the libguestfs mailing list.)): NovaException: Error mounting <RBDImage:{'name': u'7dbca304-535d-4872-939a-29f5e810f948_disk', 'format': 'raw', 'servers': ['fd00::cefc:1:6789', 'fd00::cefc:2:6789', 'fd00::
  cefc:3:6789'], 'user': None, 'password': None, 'pool': 'vms'}> with libguestfs (guestfs_launch failed.
  This usually means the libguestfs appliance failed to start or crashed.
  Do:
    export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
  and run the command again.  For further information, read:
    http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
  You can also run 'libguestfs-test-tool' and post the *complete* output
  into a bug report or message to the libguestfs mailing list.)

  This is actually being caused by the IPv6 addresses not having
  brackets, as it should be '[fd00::cefc:1]:6789'

  Even when formatted correctly, libguestfs wouldn't accept these but this was actually fixed earlier today. 
  https://github.com/libguestfs/libguestfs/commit/e047cc4da8c69f20b97965268b2b951ebbed0232 

  DNS with AAAA records would work, but it gets the addresses directly
  from ceph and has to be addresses.

  I have been able to track this down for the most part. The addresses
  are retrieved from ceph in the get_mon_addrs method and the brackets
  are removed :
  https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/storage/rbd_utils.py#n163

  This is used at here to make the dict from the error message in the get_model method:
  https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/imagebackend.py#n956

  The 'servers' variable there is the one used. As best as I've been
  able to tell, it is only used for libguestfs.

  It looks like the loop here would need to be expanded to check for IPv6 and add brackets accordingly.
  https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/imagebackend.py#n964

  I'm unsure of the best way to handle this, but since Ceph should only return IPv4 or IPv6 addresses, something like this could be done right before 964 to modify the list that is used to make 'servers'
  hosts = [ '[%s]' % k if ':' in k else k for k in hosts ]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1822689/+subscriptions