yahoo-eng-team team mailing list archive

[OpenStack Infra <1687581@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/472266
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f2d27f6a8afb62815fb6a885bd4f8ae4ed287fd3
Submitter: Jenkins
Branch:    master

commit f2d27f6a8afb62815fb6a885bd4f8ae4ed287fd3
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Thu Jun 8 09:35:42 2017 -0400

    libvirt: handle missing rbd_secret_uuid from old connection info
    
    Change Idcbada705c1d38ac5fd7c600141c2de7020eae25 in Ocata
    started preferring Cinder connection info for getting RBD auth
    values since Nova needs to be using the same settings as Cinder
    for volume auth.
    
    However, that introduced a problem for guest connections made
    before that change, where the secret_uuid might not have been
    configured on the Cinder side and that's what is stored in the
    block_device_mappings.connection_info column and is what we're
    checking in _set_auth_config_rbd. Before Ocata this wasn't a
    problem because we'd use the Nova configuration values for the
    rbd_secret_uuid if set. But since Ocata it is a problem since
    we don't consult nova.conf if auth was enabled, but not completely
    configured, on the Cinder side.
    
    So this adds a fallback check to set the secret_uuid from
    nova.conf if it wasn't set in the connection_info via Cinder
    originally. A note is also added to caution about removing
    any fallback mechanism on the nova side - something we'd
    need to consider before we could likely drop this code.
    
    Co-Authored-By: Tadas Ustinavičius <tadas@xxxxxxx>
    
    Change-Id: I6fc7108817fcd9df4a342c9dabbf14ab7911d06a
    Closes-Bug: #1687581


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1687581

Title:
  Newton > Ocata upgrade. secret_uuid empty

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) ocata series:
  In Progress

Bug description:
  Hello,
  I've bumped into issue with Nova:
  After upgrading Neutron to Ocata via openstack-ansible, I was unable to start any of the servers. 

  
  openstack server list |grep a6200dff-2010-44a2-92ca-3af303ec3577

  | a6200dff-2010-44a2-92ca-3af303ec3577 | Win10VDI-3-ephemeral |
  SHUTOFF | internal=10.0.0.17|

  openstack server start a6200dff-2010-44a2-92ca-3af303ec3577
  openstack server list |grep a6200dff-2010-44a2-92ca-3af303ec3577

  | a6200dff-2010-44a2-92ca-3af303ec3577 | Win10VDI-3-ephemeral |
  SHUTOFF | internal=10.0.0.17|

  Server is still in SHUTOFF state.
  nova.log contains the flowing entries: 
  http://paste.ubuntu.com/24465172/

  
  After digging more into problem, I've found, that  https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/volume/net.py#L65  is defined as None.
  This made me read _set_auth_config_rbd description, so I've reconfigured my cinder nodes and added `rbd_secret_uuid` line (this line was not provided in my Newton cinder configuration, but everything seemed to work fine back then).
  After reconfiguration, I've managed to create and start new machines, but machines, which were created in Newton sill failed. After some more debugging it came to me, that old machines still had no 'secret_uuid' in their connection_info dictionary:

  {u'driver_volume_type': u'rbd', u'connector': {u'wwpns':
  [u'21000024ff54b106', u'21000024ff54b107'], u'wwnns':
  [u'20000024ff54b106', u'20000024ff54b107'], u'ip': u'172.31.105.203',
  u'initiator': u'iqn.1993-08.org.debian:01:b7f25b95aff0', u'platform':
  u'x86_64', u'host': u'ostack-ibm3', u'do_local_attach': False,
  u'os_type': u'linux2', u'multipath': False}, u'serial':
  u'87c5d950-c212-4e9c-9d13-6a583fed9e62', u'data': {u'secret_type':
  u'ceph', u'name': u'cinder-volumes/volume-87c5d950-c212-4e9c-
  9d13-6a583fed9e62', u'encrypted': False, u'cluster_name': u'ceph',
  u'secret_uuid': None, u'qos_specs': None, u'hosts': [u'172.31.104.1',
  u'172.31.104.2', u'172.31.104.3'], u'volume_id': u'87c5d950-c212-4e9c-
  9d13-6a583fed9e62', u'auth_enabled': True, u'access_mode': u'rw',
  u'auth_username': u'ostack', u'ports': [u'6789', u'6789', u'6789']}}

  
  It seems, that Ocata requires secret_uuid to be provided in database. Machines form Newton release did not contain that entry:

  Newton :

  | {"driver_volume_type": "rbd", "connector": {"wwpns": ["21000024ff54b106", "21000024ff54b107"], "wwnns": ["20000024ff54b106", "20000024ff54b107"], "ip": "172.31.105.203", "initiator": "iqn.1993-08.org.debian:01:b7f25b95aff0", "platform": "x86_64", "host": "ostack-ibm3", "do_local_attach": false, "os_type": "linux2", "multipath": false}, "serial": "87326e73-1e9a-4589-9140-985950e93068", "data": {"secret_type": "ceph", "name": "cinder-volumes/volume-87326e73-1e9a-4589-9140-985950e93068", "encrypted": false, "cluster_name": "ceph", "secret_uuid": null, "qos_specs": null, "hosts": ["172.31.104.1", "172.31.104.2", "172.31.104.3"], "volume_id": "87326e73-1e9a-4589-9140-985950e93068", "auth_enabled": true, "access_mode": "rw", "auth_username": "ostack", "ports": ["6789", "6789", "6789"]}}                                                                                           
  Ocata: 
                                               |
  | {"driver_volume_type": "rbd", "connector": {"wwpns": ["21000024ff54b106", "21000024ff54b107"], "wwnns": ["20000024ff54b106", "20000024ff54b107"], "ip": "172.31.105.203", "initiator": "iqn.1993-08.org.debian:01:b7f25b95aff0", "platform": "x86_64", "host": "ostack-ibm3", "do_local_attach": false, "os_type": "linux2", "multipath": false}, "serial": "565ba4ab-2f8a-40fd-ae82-9346414ceb15", "data": {"secret_type": "ceph", "name": "cinder-volumes/volume-565ba4ab-2f8a-40fd-ae82-9346414ceb15", "encrypted": false, "cluster_name": "ceph", "secret_uuid": "a11833c5-9403-4423-8a26-111222333444", "qos_specs": null, "hosts": ["172.31.104.1", "172.31.104.2", "172.31.104.3"], "volume_id": "565ba4ab-2f8a-40fd-ae82-9346414ceb15", "auth_enabled": true, "access_mode": "rw", "auth_username": "ostack", "ports": ["6789", "6789", "6789"]}}  


  So, is there any way to refresh database to fix this, or should I refresh block_device_mapping table by hand?  
  I'm not sure if this is a bug, perhaps there is need to put some warning about this in upgrade documentation? 
  Thank you.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1687581/+subscriptions