yahoo-eng-team team mailing list archive

[Anh Tran <anhtt@xxxxxxxxxxxxxx>]

Public bug reported:

I have 3 controller nodes running HA active/active, using KVM hypervisor
and Maria cluster as shared database. The system was deployed by
Devstack Mitaka version on virtual machines which was created by virt-
manager.

I have upgraded Keytone to N version, then I tried to Rolling Upgrade Nova from M to N version folowed:
http://docs.openstack.org/developer/nova/upgrade.html#rolling-upgrade-process

The document said that:
Using the newly installed nova code, run the DB sync. (nova-manage db sync; nova-manage api_db sync). These schema change operations should have minimal or no effect on performance, and should not cause any operations to fail.

However, during the sync database, I cannot create the VM. Nova raise
that:

ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.DBDeadlock'> (HTTP 500) (Request-ID: req-b5c82715-6306-4f6d-972b-3387015da12c)

Full log here: http://paste.openstack.org/show/588365/

After finishing the sync process, I can create VM as well.


==Reproduce==

# Controller1:
1. Stop all nova service, except nova api (n-api).

2. Upgrade source code:
$ cd /opt/stack/nova/
$ git checkout remotes/origin/stable/newton
$ git checkout -b stable/newton remotes/origin/stable/newton
$ git pull
$ sudo -E pip install -r requirements.txt --upgrade

3. Downgrade some packages dependency (because I used --upgrade as above)
$ sudo pip uninstall oslo.messaging
$ sudo pip uninstall kombu
$ sudo pip uninstall cffi
$ sudo -E pip install oslo.messaging==5.10.0
$ sudo -E pip install kombu==3.0.35
$ sudo -E pip install cffi==1.5.2

4. Update /etc/nova/nova.conf:
[upgrade_levels]
compute = auto

5. Sync DB
$ nova-manage db sync
$ nova-manage api_db sync

6. During the Sync DB, try to create VM, execute on controller 2 and 3 (not concurrency):
$ nova boot --flavor m1.nano --image 21ffa33b-e9eb-43f4-aa73-ceb8f2cbc6fc --nic net-name=net1 VM_test
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.DBDeadlock'> (HTTP 500) (Request-ID: req-b5c82715-6306-4f6d-972b-3387015da12c)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1640164

Title:
  Rolling upgrade M to N:  DBDeadlock Error when create instance during
  sync database

Status in OpenStack Compute (nova):
  New

Bug description:
  I have 3 controller nodes running HA active/active, using KVM
  hypervisor and Maria cluster as shared database. The system was
  deployed by Devstack Mitaka version on virtual machines which was
  created by virt-manager.

  I have upgraded Keytone to N version, then I tried to Rolling Upgrade Nova from M to N version folowed:
  http://docs.openstack.org/developer/nova/upgrade.html#rolling-upgrade-process

  The document said that:
  Using the newly installed nova code, run the DB sync. (nova-manage db sync; nova-manage api_db sync). These schema change operations should have minimal or no effect on performance, and should not cause any operations to fail.

  However, during the sync database, I cannot create the VM. Nova raise
  that:

  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_db.exception.DBDeadlock'> (HTTP 500) (Request-ID: req-b5c82715-6306-4f6d-972b-3387015da12c)

  Full log here: http://paste.openstack.org/show/588365/

  After finishing the sync process, I can create VM as well.

  
  ==Reproduce==

  # Controller1:
  1. Stop all nova service, except nova api (n-api).

  2. Upgrade source code:
  $ cd /opt/stack/nova/
  $ git checkout remotes/origin/stable/newton
  $ git checkout -b stable/newton remotes/origin/stable/newton
  $ git pull
  $ sudo -E pip install -r requirements.txt --upgrade

  3. Downgrade some packages dependency (because I used --upgrade as above)
  $ sudo pip uninstall oslo.messaging
  $ sudo pip uninstall kombu
  $ sudo pip uninstall cffi
  $ sudo -E pip install oslo.messaging==5.10.0
  $ sudo -E pip install kombu==3.0.35
  $ sudo -E pip install cffi==1.5.2

  4. Update /etc/nova/nova.conf:
  [upgrade_levels]
  compute = auto

  5. Sync DB
  $ nova-manage db sync
  $ nova-manage api_db sync

  6. During the Sync DB, try to create VM, execute on controller 2 and 3 (not concurrency):
  $ nova boot --flavor m1.nano --image 21ffa33b-e9eb-43f4-aa73-ceb8f2cbc6fc --nic net-name=net1 VM_test
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_db.exception.DBDeadlock'> (HTTP 500) (Request-ID: req-b5c82715-6306-4f6d-972b-3387015da12c)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1640164/+subscriptions