yahoo-eng-team team mailing list archive

[Doug Hellmann <doug@xxxxxxxxxxxxxxxx>]

** Changed in: oslo.messaging
       Status: Fix Committed => Fix Released

** Changed in: oslo.messaging
    Milestone: None => 1.8.1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1430984

Title:
  Recent RPC namespacing breaks rolling upgrades

Status in OpenStack Neutron (virtual network service):
  Fix Committed
Status in Messaging API for OpenStack:
  Fix Released

Bug description:
  Several patches merged as a part of:
  https://review.openstack.org/#/q/status:merged+project:openstack/neutron+branch:master+topic:bp/rpc-docs-and-namespaces,n,z

  Broke Neutron rolling upgrade (Specifically: Upgrading the server(s)
  before the agents or vice versa). This was done knowingly and
  discussed in the spec process. While we don't test a rolling upgrade
  scenario, there is no reason to break it knowingly. I've spoken to
  operators that have successfully performed such an upgrade from I to J
  and it will be very surprising to them if the same doesn't work from J
  to K.

  The breakage comes from the introduction of RPC namespaces, a very
  useful concept of putting RPC endpoints in separate namespaces. i.e.
  you may place the same method name listening in the same process if it
  belongs to different namespaces.

  Possible solutions:
  Have the server listen on both the new namespaces, and in the root namespace. However, this effectively brings all such methods into one big namespace, so this kind of defeats the purpose of namespacing. We could delay by making a change in Oslo messaging where if a new and optional backwards_compatibility flag is passed in to a target along with a namespace, then the dispatcher will check against the namespace as well as the root namespace, and we simply stop passing the flag in the L cycle (This means that we only support rolling upgrades from version N to N+1). In order to support a scenario where an agent is upgraded before a server, then even with the proposed solution, all of the K agents would have to implement a fallback.

  Testing:
  I've been working on basic RPC tests:
  https://review.openstack.org/#/q/status:open+project:openstack/neutron+branch:master+topic:rpc_tests,n,z

  But I don't think such a framework will allow us to test rolling
  upgrades. I can't think of an alternative to actually performing one
  and seeing what happens (A spin on the grenade job).

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1430984/+subscriptions