← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #63728

[Bug 1687581] [NEW] Newton > Ocata upgrade. secret_uuid empty

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Hello,
I've bumped into issue with Nova:
After upgrading Neutron to Ocata via openstack-ansible, I was unable to start any of the servers. 


openstack server list |grep a6200dff-2010-44a2-92ca-3af303ec3577

| a6200dff-2010-44a2-92ca-3af303ec3577 | Win10VDI-3-ephemeral | SHUTOFF
| internal=10.0.0.17|

openstack server start a6200dff-2010-44a2-92ca-3af303ec3577
openstack server list |grep a6200dff-2010-44a2-92ca-3af303ec3577

| a6200dff-2010-44a2-92ca-3af303ec3577 | Win10VDI-3-ephemeral | SHUTOFF
| internal=10.0.0.17|

Server is still in SHUTOFF state.
nova.log contains the flowing entries: 
http://paste.ubuntu.com/24465172/


After digging more into problem, I've found, that  https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/volume/net.py#L65  is defined as None.
This made me read _set_auth_config_rbd description, so I've reconfigured my cinder nodes and added `rbd_secret_uuid` line (this line was not provided in my Newton cinder configuration, but everything seemed to work fine back then).
After reconfiguration, I've managed to create and start new machines, but machines, which were created in Newton sill failed. After some more debugging it came to me, that old machines still had no 'secret_uuid' in their connection_info dictionary:

{u'driver_volume_type': u'rbd', u'connector': {u'wwpns':
[u'21000024ff54b106', u'21000024ff54b107'], u'wwnns':
[u'20000024ff54b106', u'20000024ff54b107'], u'ip': u'172.31.105.203',
u'initiator': u'iqn.1993-08.org.debian:01:b7f25b95aff0', u'platform':
u'x86_64', u'host': u'ostack-ibm3', u'do_local_attach': False,
u'os_type': u'linux2', u'multipath': False}, u'serial': u'87c5d950-c212
-4e9c-9d13-6a583fed9e62', u'data': {u'secret_type': u'ceph', u'name': u
'cinder-volumes/volume-87c5d950-c212-4e9c-9d13-6a583fed9e62',
u'encrypted': False, u'cluster_name': u'ceph', u'secret_uuid': None,
u'qos_specs': None, u'hosts': [u'172.31.104.1', u'172.31.104.2',
u'172.31.104.3'], u'volume_id': u'87c5d950-c212-4e9c-9d13-6a583fed9e62',
u'auth_enabled': True, u'access_mode': u'rw', u'auth_username':
u'ostack', u'ports': [u'6789', u'6789', u'6789']}}


It seems, that Ocata requires secret_uuid to be provided in database. Machines form Newton release did not contain that entry:

Newton :

| {"driver_volume_type": "rbd", "connector": {"wwpns": ["21000024ff54b106", "21000024ff54b107"], "wwnns": ["20000024ff54b106", "20000024ff54b107"], "ip": "172.31.105.203", "initiator": "iqn.1993-08.org.debian:01:b7f25b95aff0", "platform": "x86_64", "host": "ostack-ibm3", "do_local_attach": false, "os_type": "linux2", "multipath": false}, "serial": "87326e73-1e9a-4589-9140-985950e93068", "data": {"secret_type": "ceph", "name": "cinder-volumes/volume-87326e73-1e9a-4589-9140-985950e93068", "encrypted": false, "cluster_name": "ceph", "secret_uuid": null, "qos_specs": null, "hosts": ["172.31.104.1", "172.31.104.2", "172.31.104.3"], "volume_id": "87326e73-1e9a-4589-9140-985950e93068", "auth_enabled": true, "access_mode": "rw", "auth_username": "ostack", "ports": ["6789", "6789", "6789"]}}                                                                                           
Ocata: 
                                             |
| {"driver_volume_type": "rbd", "connector": {"wwpns": ["21000024ff54b106", "21000024ff54b107"], "wwnns": ["20000024ff54b106", "20000024ff54b107"], "ip": "172.31.105.203", "initiator": "iqn.1993-08.org.debian:01:b7f25b95aff0", "platform": "x86_64", "host": "ostack-ibm3", "do_local_attach": false, "os_type": "linux2", "multipath": false}, "serial": "565ba4ab-2f8a-40fd-ae82-9346414ceb15", "data": {"secret_type": "ceph", "name": "cinder-volumes/volume-565ba4ab-2f8a-40fd-ae82-9346414ceb15", "encrypted": false, "cluster_name": "ceph", "secret_uuid": "a11833c5-9403-4423-8a26-111222333444", "qos_specs": null, "hosts": ["172.31.104.1", "172.31.104.2", "172.31.104.3"], "volume_id": "565ba4ab-2f8a-40fd-ae82-9346414ceb15", "auth_enabled": true, "access_mode": "rw", "auth_username": "ostack", "ports": ["6789", "6789", "6789"]}}  


So, is there any way to refresh database to fix this, or should I refresh block_device_mapping table by hand?  
I'm not sure if this is a bug, perhaps there is need to put some warning about this in upgrade documentation? 
Thank you.

** Affects: nova
     Importance: Undecided
         Status: New

** Summary changed:

- Newton > Ocata upgrace. secret_uuid empty
+ Newton > Ocata upgrade. secret_uuid empty

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1687581

Title:
  Newton > Ocata upgrade. secret_uuid empty

Status in OpenStack Compute (nova):
  New

Bug description:
  Hello,
  I've bumped into issue with Nova:
  After upgrading Neutron to Ocata via openstack-ansible, I was unable to start any of the servers. 

  
  openstack server list |grep a6200dff-2010-44a2-92ca-3af303ec3577

  | a6200dff-2010-44a2-92ca-3af303ec3577 | Win10VDI-3-ephemeral |
  SHUTOFF | internal=10.0.0.17|

  openstack server start a6200dff-2010-44a2-92ca-3af303ec3577
  openstack server list |grep a6200dff-2010-44a2-92ca-3af303ec3577

  | a6200dff-2010-44a2-92ca-3af303ec3577 | Win10VDI-3-ephemeral |
  SHUTOFF | internal=10.0.0.17|

  Server is still in SHUTOFF state.
  nova.log contains the flowing entries: 
  http://paste.ubuntu.com/24465172/

  
  After digging more into problem, I've found, that  https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/volume/net.py#L65  is defined as None.
  This made me read _set_auth_config_rbd description, so I've reconfigured my cinder nodes and added `rbd_secret_uuid` line (this line was not provided in my Newton cinder configuration, but everything seemed to work fine back then).
  After reconfiguration, I've managed to create and start new machines, but machines, which were created in Newton sill failed. After some more debugging it came to me, that old machines still had no 'secret_uuid' in their connection_info dictionary:

  {u'driver_volume_type': u'rbd', u'connector': {u'wwpns':
  [u'21000024ff54b106', u'21000024ff54b107'], u'wwnns':
  [u'20000024ff54b106', u'20000024ff54b107'], u'ip': u'172.31.105.203',
  u'initiator': u'iqn.1993-08.org.debian:01:b7f25b95aff0', u'platform':
  u'x86_64', u'host': u'ostack-ibm3', u'do_local_attach': False,
  u'os_type': u'linux2', u'multipath': False}, u'serial':
  u'87c5d950-c212-4e9c-9d13-6a583fed9e62', u'data': {u'secret_type':
  u'ceph', u'name': u'cinder-volumes/volume-87c5d950-c212-4e9c-
  9d13-6a583fed9e62', u'encrypted': False, u'cluster_name': u'ceph',
  u'secret_uuid': None, u'qos_specs': None, u'hosts': [u'172.31.104.1',
  u'172.31.104.2', u'172.31.104.3'], u'volume_id': u'87c5d950-c212-4e9c-
  9d13-6a583fed9e62', u'auth_enabled': True, u'access_mode': u'rw',
  u'auth_username': u'ostack', u'ports': [u'6789', u'6789', u'6789']}}

  
  It seems, that Ocata requires secret_uuid to be provided in database. Machines form Newton release did not contain that entry:

  Newton :

  | {"driver_volume_type": "rbd", "connector": {"wwpns": ["21000024ff54b106", "21000024ff54b107"], "wwnns": ["20000024ff54b106", "20000024ff54b107"], "ip": "172.31.105.203", "initiator": "iqn.1993-08.org.debian:01:b7f25b95aff0", "platform": "x86_64", "host": "ostack-ibm3", "do_local_attach": false, "os_type": "linux2", "multipath": false}, "serial": "87326e73-1e9a-4589-9140-985950e93068", "data": {"secret_type": "ceph", "name": "cinder-volumes/volume-87326e73-1e9a-4589-9140-985950e93068", "encrypted": false, "cluster_name": "ceph", "secret_uuid": null, "qos_specs": null, "hosts": ["172.31.104.1", "172.31.104.2", "172.31.104.3"], "volume_id": "87326e73-1e9a-4589-9140-985950e93068", "auth_enabled": true, "access_mode": "rw", "auth_username": "ostack", "ports": ["6789", "6789", "6789"]}}                                                                                           
  Ocata: 
                                               |
  | {"driver_volume_type": "rbd", "connector": {"wwpns": ["21000024ff54b106", "21000024ff54b107"], "wwnns": ["20000024ff54b106", "20000024ff54b107"], "ip": "172.31.105.203", "initiator": "iqn.1993-08.org.debian:01:b7f25b95aff0", "platform": "x86_64", "host": "ostack-ibm3", "do_local_attach": false, "os_type": "linux2", "multipath": false}, "serial": "565ba4ab-2f8a-40fd-ae82-9346414ceb15", "data": {"secret_type": "ceph", "name": "cinder-volumes/volume-565ba4ab-2f8a-40fd-ae82-9346414ceb15", "encrypted": false, "cluster_name": "ceph", "secret_uuid": "a11833c5-9403-4423-8a26-111222333444", "qos_specs": null, "hosts": ["172.31.104.1", "172.31.104.2", "172.31.104.3"], "volume_id": "565ba4ab-2f8a-40fd-ae82-9346414ceb15", "auth_enabled": true, "access_mode": "rw", "auth_username": "ostack", "ports": ["6789", "6789", "6789"]}}  


  So, is there any way to refresh database to fix this, or should I refresh block_device_mapping table by hand?  
  I'm not sure if this is a bug, perhaps there is need to put some warning about this in upgrade documentation? 
  Thank you.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1687581/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next