yahoo-eng-team team mailing list archive

[Anh Tran <anhtt@xxxxxxxxxxxxxx>]

Public bug reported:

I have 3 Controller nodes running HA active/active mode. Using Mysql-server as shared database.
After upgrade Controller1, I start it to handle the request to make the system no downtime.
But when a request is handling by Controller1, an error happended: "There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set". Keystone raise that: KeyError: 'is_domain' 

How to reproduce:
Follow this guide: http://docs.openstack.org/developer/keystone/upgrading.html#upgrading-without-downtime

# Controller1
$ sudo service apache2 stop

$ cd /opt/stack/keystone/
$ git checkout remotes/origin/stable/newton
$ git checkout -b stable/newton remotes/origin/stable/newton
$ sudo pip install -r requirements.txt --upgrade

$ keystone-manage doctor
$ keystone-manage db_sync --expand
$ keystone-manage db_sync --migrate
$ sudo python setup.py install
$ sudo service apache2 start

# Controller2 or any openstack clients
$ for i in {1..10}; do openstack neutron network list; done
...
503 Service Unavailable
The server is currently unavailable. Please try again at a later time
...

Full log in kestone here: http://paste.openstack.org/show/584107/

After I upgraded all 3 Controller nodes follow the same above steps
except upgrading db, the error never occurs again.

At step 9 in the guideline: "Upgrade all keystone nodes to the next release, and restart them one at a time..."
I think we will have downtime in this process. So I tried to upgrade controller1 first, then make it online to ensure that the system have not downtime.

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1630259

Title:
  Rolling upgrade does not work well in Newton release

Status in OpenStack Identity (keystone):
  New

Bug description:
  I have 3 Controller nodes running HA active/active mode. Using Mysql-server as shared database.
  After upgrade Controller1, I start it to handle the request to make the system no downtime.
  But when a request is handling by Controller1, an error happended: "There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set". Keystone raise that: KeyError: 'is_domain' 

  How to reproduce:
  Follow this guide: http://docs.openstack.org/developer/keystone/upgrading.html#upgrading-without-downtime

  # Controller1
  $ sudo service apache2 stop

  $ cd /opt/stack/keystone/
  $ git checkout remotes/origin/stable/newton
  $ git checkout -b stable/newton remotes/origin/stable/newton
  $ sudo pip install -r requirements.txt --upgrade

  $ keystone-manage doctor
  $ keystone-manage db_sync --expand
  $ keystone-manage db_sync --migrate
  $ sudo python setup.py install
  $ sudo service apache2 start

  # Controller2 or any openstack clients
  $ for i in {1..10}; do openstack neutron network list; done
  ...
  503 Service Unavailable
  The server is currently unavailable. Please try again at a later time
  ...

  Full log in kestone here: http://paste.openstack.org/show/584107/

  After I upgraded all 3 Controller nodes follow the same above steps
  except upgrading db, the error never occurs again.

  At step 9 in the guideline: "Upgrade all keystone nodes to the next release, and restart them one at a time..."
  I think we will have downtime in this process. So I tried to upgrade controller1 first, then make it online to ensure that the system have not downtime.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1630259/+subscriptions