yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1481557] [NEW] kilo default cinder behavior changed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: David Medberry <1481557@xxxxxxxxxxxxxxxxxx>
Date: Wed, 05 Aug 2015 02:53:26 -0000
Reply-to: Bug 1481557 <1481557@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Prior to Kilo, not having a:
[cinder]
os_region_name=SET

set meant that nova would look in the region the nova call went to. As
of Kilo, it appears to do an endpoint list and pull the first cinder
endpoint it finds (even if it is in a different region.)

I've not yet reproduced this in devstack but a multi-region devstack
should see the right behavior prior to kilo and the wrong behavior in
kilo (and presumably in Liberty.)

This has a VERY STRONG OPERATIONS IMPACT due to the difficulty in
debugging a cross region error like this (it's extraordinarily opaque).

The simplest reproducer is to have a multi region openstack (for both
nova and cinder) and create instances and volumes in both regions. In
each region, try and attach a volume to a running instance. One side
(whichever is uuid-numerically-lower) will likely succeed and the one
that is numerically higher will fail (as it will try and find the volume
in the wrong region.

The workaround is straight-forward--set 
[cinder]
os_region_name=REGION
in the nova.conf file but as this is a behavior change, it should have been called out in the release notes.

Running:
openstack kilo 2015.1.0 (nova and cinder) in a multi (well two) region env.

distro info:

[STAGING] medberry@chrcnc02-control-003:~$ dpkg -l \*nova\* \*cinder\* \*keystone\* \*neutron\* |cat |grep ii
ii  cinder-api                           1:2015.1.0-0ubuntu1~cloud0            all          Cinder storage service - API server
ii  cinder-backup                        1:2015.1.0-0ubuntu1~cloud0            all          Cinder storage service - Scheduler server
ii  cinder-common                        1:2015.1.0-0ubuntu1~cloud0            all          Cinder storage service - common files
ii  cinder-scheduler                     1:2015.1.0-0ubuntu1~cloud0            all          Cinder storage service - Scheduler server
ii  cinder-volume                        1:2015.1.0-0ubuntu1~cloud0            all          Cinder storage service - Volume server
ii  neutron-common                       1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - common
ii  neutron-dhcp-agent                   1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - DHCP agent
ii  neutron-l3-agent                     1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - l3 agent
ii  neutron-metadata-agent               1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - metadata agent
ii  neutron-plugin-ml2                   1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - ML2 plugin
ii  neutron-plugin-openvswitch-agent     1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii  neutron-server                       1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - server
ii  nova-api                             1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - API frontend
ii  nova-cert                            1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - certificate management
ii  nova-common                          1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - common files
ii  nova-conductor                       1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - conductor service
ii  nova-consoleauth                     1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - Console Authenticator
ii  nova-novncproxy                      1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - NoVNC proxy
ii  nova-objectstore                     1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - object store
ii  nova-scheduler                       1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - virtual machine scheduler
ii  python-cinder                        1:2015.1.0-0ubuntu1~cloud0            all          Cinder Python libraries
ii  python-cinderclient                  1:1.1.1-0ubuntu1~cloud0               all          python bindings to the OpenStack Volume API
ii  python-keystone                      1:2015.1.0-0ubuntu1~cloud0            all          OpenStack identity service - Python library
ii  python-keystoneclient                1:1.2.0-0ubuntu1~cloud0               all          Client library for OpenStack Identity API
ii  python-keystonemiddleware            1.5.0-0ubuntu1~cloud0                 all          Middleware for OpenStack Identity (Keystone) - Python 2.x
ii  python-neutron                       1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - Python library
ii  python-neutron-fwaas                 2015.1.0-0ubuntu1~cloud0              all          Firewall-as-a-Service driver for OpenStack Neutron
ii  python-neutron-lbaas                 1:2015.1.0-0ubuntu1~cloud0            all          Loadbalancer-as-a-Service driver for OpenStack Neutron
ii  python-neutronclient                 1:2.3.11-0ubuntu1~cloud0              all          client - Neutron is a virtual network service for Openstack
ii  python-nova                          1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute Python libraries
ii  python-novaclient                    1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API
[STAGING] medberry@chrcnc02-control-003:~$

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: ops

** Tags added: ops

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1481557

Title:
  kilo default cinder behavior changed

Status in OpenStack Compute (nova):
  New

Bug description:
  Prior to Kilo, not having a:
  [cinder]
  os_region_name=SET

  set meant that nova would look in the region the nova call went to. As
  of Kilo, it appears to do an endpoint list and pull the first cinder
  endpoint it finds (even if it is in a different region.)

  I've not yet reproduced this in devstack but a multi-region devstack
  should see the right behavior prior to kilo and the wrong behavior in
  kilo (and presumably in Liberty.)

  This has a VERY STRONG OPERATIONS IMPACT due to the difficulty in
  debugging a cross region error like this (it's extraordinarily
  opaque).

  The simplest reproducer is to have a multi region openstack (for both
  nova and cinder) and create instances and volumes in both regions. In
  each region, try and attach a volume to a running instance. One side
  (whichever is uuid-numerically-lower) will likely succeed and the one
  that is numerically higher will fail (as it will try and find the
  volume in the wrong region.

  The workaround is straight-forward--set 
  [cinder]
  os_region_name=REGION
  in the nova.conf file but as this is a behavior change, it should have been called out in the release notes.

  Running:
  openstack kilo 2015.1.0 (nova and cinder) in a multi (well two) region env.

  distro info:

  [STAGING] medberry@chrcnc02-control-003:~$ dpkg -l \*nova\* \*cinder\* \*keystone\* \*neutron\* |cat |grep ii
  ii  cinder-api                           1:2015.1.0-0ubuntu1~cloud0            all          Cinder storage service - API server
  ii  cinder-backup                        1:2015.1.0-0ubuntu1~cloud0            all          Cinder storage service - Scheduler server
  ii  cinder-common                        1:2015.1.0-0ubuntu1~cloud0            all          Cinder storage service - common files
  ii  cinder-scheduler                     1:2015.1.0-0ubuntu1~cloud0            all          Cinder storage service - Scheduler server
  ii  cinder-volume                        1:2015.1.0-0ubuntu1~cloud0            all          Cinder storage service - Volume server
  ii  neutron-common                       1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - common
  ii  neutron-dhcp-agent                   1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - DHCP agent
  ii  neutron-l3-agent                     1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - l3 agent
  ii  neutron-metadata-agent               1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - metadata agent
  ii  neutron-plugin-ml2                   1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - ML2 plugin
  ii  neutron-plugin-openvswitch-agent     1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
  ii  neutron-server                       1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - server
  ii  nova-api                             1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - API frontend
  ii  nova-cert                            1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - certificate management
  ii  nova-common                          1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - common files
  ii  nova-conductor                       1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - conductor service
  ii  nova-consoleauth                     1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                      1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - NoVNC proxy
  ii  nova-objectstore                     1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - object store
  ii  nova-scheduler                       1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - virtual machine scheduler
  ii  python-cinder                        1:2015.1.0-0ubuntu1~cloud0            all          Cinder Python libraries
  ii  python-cinderclient                  1:1.1.1-0ubuntu1~cloud0               all          python bindings to the OpenStack Volume API
  ii  python-keystone                      1:2015.1.0-0ubuntu1~cloud0            all          OpenStack identity service - Python library
  ii  python-keystoneclient                1:1.2.0-0ubuntu1~cloud0               all          Client library for OpenStack Identity API
  ii  python-keystonemiddleware            1.5.0-0ubuntu1~cloud0                 all          Middleware for OpenStack Identity (Keystone) - Python 2.x
  ii  python-neutron                       1:2015.1.0-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - Python library
  ii  python-neutron-fwaas                 2015.1.0-0ubuntu1~cloud0              all          Firewall-as-a-Service driver for OpenStack Neutron
  ii  python-neutron-lbaas                 1:2015.1.0-0ubuntu1~cloud0            all          Loadbalancer-as-a-Service driver for OpenStack Neutron
  ii  python-neutronclient                 1:2.3.11-0ubuntu1~cloud0              all          client - Neutron is a virtual network service for Openstack
  ii  python-nova                          1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute Python libraries
  ii  python-novaclient                    1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API
  [STAGING] medberry@chrcnc02-control-003:~$

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1481557/+subscriptions
Follow ups

[Bug 1481557] Re: kilo default cinder behavior changed
From: Launchpad Bug Tracker, 2015-10-05