yahoo-eng-team team mailing list archive

[Maciej Szankin <maciej@xxxxxxxxxx>]

** Changed in: nova
       Status: In Progress => Won't Fix

** Changed in: nova
       Status: Won't Fix => New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1640164

Title:
  Rolling upgrade M to N:  DBDeadlock Error when create instance during
  sync database

Status in OpenStack Compute (nova):
  New

Bug description:
  I have 3 controller nodes running HA active/active, using KVM
  hypervisor and Maria cluster as shared database. The system was
  deployed by Devstack Mitaka version on virtual machines which was
  created by virt-manager.

  I have upgraded Keytone to N version, then I tried to Rolling Upgrade Nova from M to N version folowed:
  http://docs.openstack.org/developer/nova/upgrade.html#rolling-upgrade-process

  The document said that:
  Using the newly installed nova code, run the DB sync. (nova-manage db sync; nova-manage api_db sync). These schema change operations should have minimal or no effect on performance, and should not cause any operations to fail.

  However, during the sync database, I cannot create the VM. Nova raise
  that:

  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_db.exception.DBDeadlock'> (HTTP 500) (Request-ID: req-b5c82715-6306-4f6d-972b-3387015da12c)

  Full log here: http://paste.openstack.org/show/588365/

  After finishing the sync process, I can create VM as well.

  
  ==Reproduce==

  # Controller1:
  1. Stop all nova service, except nova api (n-api).

  2. Upgrade source code:
  $ cd /opt/stack/nova/
  $ git checkout remotes/origin/stable/newton
  $ git checkout -b stable/newton remotes/origin/stable/newton
  $ git pull
  $ sudo -E pip install -r requirements.txt --upgrade

  3. Downgrade some packages dependency (because I used --upgrade as above)
  $ sudo pip uninstall oslo.messaging
  $ sudo pip uninstall kombu
  $ sudo pip uninstall cffi
  $ sudo -E pip install oslo.messaging==5.10.0
  $ sudo -E pip install kombu==3.0.35
  $ sudo -E pip install cffi==1.5.2

  4. Update /etc/nova/nova.conf:
  [upgrade_levels]
  compute = auto

  5. Sync DB
  $ nova-manage db sync
  $ nova-manage api_db sync

  6. During the Sync DB, try to create VM, execute on controller 2 and 3 (not concurrency):
  $ nova boot --flavor m1.nano --image 21ffa33b-e9eb-43f4-aa73-ceb8f2cbc6fc --nic net-name=net1 VM_test
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_db.exception.DBDeadlock'> (HTTP 500) (Request-ID: req-b5c82715-6306-4f6d-972b-3387015da12c)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1640164/+subscriptions