yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1582224] [NEW] Post liberty upgrade neutron-server wont start - seemingly related to ml2_vlan_allocations table

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ross Martyn <rossmartyn04@xxxxxxxxx>
Date: Mon, 16 May 2016 13:10:11 -0000
Reply-to: Bug 1582224 <1582224@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Environment:
ii  neutron-plugin-ml2               2:7.0.3-0ubuntu1~cloud0               all          Neutron is a virtual network service for Openstack - ML2 plugin
ii  neutron-server                   2:7.0.3-0ubuntu1~cloud0               all          Neutron is a virtual network service for Openstack - server

:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.4 LTS
Release:	14.04
Codename:	trusty
Kernel:         3.18.26-x1-64
}}}

After upgrading Kilo to Liberty (using Ubuntu Cloud Archive Packages).
Neutron server fails to start.

We have been able to trace this back to the following query.

{{{
[SQL:
u'SELECT ml2_vlan_allocations.physical_network AS ml2_vlan_allocations_physical_network, ml2_vlan_allocations.vlan_id AS ml2_vlan_allocations_vlan_id, ml2_vlan_
allocations.allocated AS ml2_vlan_allocations_allocated \nFROM ml2_vlan_allocations FOR UPDATE']
}}}

It appears that neutron is locking the MySQL database, and a little
later, MySQL is dropping the lock causing the following:

{{{
neutron [-] DBError: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction')
}}}

Example of where Neutron Server is stuck starting up:

{{{
2016-05-16 12:39:07.859 13463 INFO neutron.manager [-] Loading core plugin: ml2
2016-05-16 12:39:07.960 13463 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['flat', 'gre', 'vlan']
2016-05-16 12:39:07.963 13463 INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['external']
2016-05-16 12:39:07.966 13463 INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: {'vlan': [(300, 4000)]}
2016-05-16 12:39:07.972 13463 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'vlan', 'gre']
2016-05-16 12:39:07.973 13463 INFO neutron.plugins.ml2.managers [-] Registered types: ['flat', 'vlan', 'gre']
2016-05-16 12:39:07.973 13463 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['gre']
2016-05-16 12:39:07.974 13463 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: []
2016-05-16 12:39:07.974 13463 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: []
2016-05-16 12:39:07.975 13463 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: []
2016-05-16 12:39:07.975 13463 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['openvswitch']
2016-05-16 12:39:07.977 13463 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['openvswitch']
2016-05-16 12:39:07.978 13463 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['openvswitch']
2016-05-16 12:39:08.062 13463 DEBUG neutron.callbacks.manager [-] Subscribe: <bound method Ml2Plugin.validate_network_rbac_policy_change of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy before_create subscribe /usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
2016-05-16 12:39:08.062 13463 DEBUG neutron.callbacks.manager [-] Subscribe: <bound method Ml2Plugin.validate_network_rbac_policy_change of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy before_update subscribe /usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
2016-05-16 12:39:08.063 13463 DEBUG neutron.callbacks.manager [-] Subscribe: <bound method Ml2Plugin.validate_network_rbac_policy_change of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy before_delete subscribe /usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
2016-05-16 12:39:08.064 13463 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat'
2016-05-16 12:39:08.064 13463 INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete
2016-05-16 12:39:08.065 13463 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'vlan'
2016-05-16 12:39:08.115 13463 DEBUG oslo_db.sqlalchemy.engines [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
}}}

We are able to use the following to populate a fresh database, then
neutron will start.

{{{
su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron
}}}

However as you can imagine its quite important that we are able to
upgrade the current database!

Appreciate any help, or links to previous related bugs that may improve
our situation.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1582224

Title:
  Post liberty upgrade neutron-server wont start - seemingly related to
  ml2_vlan_allocations table

Status in neutron:
  New

Bug description:
  Environment:
  ii  neutron-plugin-ml2               2:7.0.3-0ubuntu1~cloud0               all          Neutron is a virtual network service for Openstack - ML2 plugin
  ii  neutron-server                   2:7.0.3-0ubuntu1~cloud0               all          Neutron is a virtual network service for Openstack - server

  :~# lsb_release -a
  No LSB modules are available.
  Distributor ID:	Ubuntu
  Description:	Ubuntu 14.04.4 LTS
  Release:	14.04
  Codename:	trusty
  Kernel:         3.18.26-x1-64
  }}}

  After upgrading Kilo to Liberty (using Ubuntu Cloud Archive Packages).
  Neutron server fails to start.

  We have been able to trace this back to the following query.

  {{{
  [SQL:
  u'SELECT ml2_vlan_allocations.physical_network AS ml2_vlan_allocations_physical_network, ml2_vlan_allocations.vlan_id AS ml2_vlan_allocations_vlan_id, ml2_vlan_
  allocations.allocated AS ml2_vlan_allocations_allocated \nFROM ml2_vlan_allocations FOR UPDATE']
  }}}

  It appears that neutron is locking the MySQL database, and a little
  later, MySQL is dropping the lock causing the following:

  {{{
  neutron [-] DBError: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction')
  }}}

  Example of where Neutron Server is stuck starting up:

  {{{
  2016-05-16 12:39:07.859 13463 INFO neutron.manager [-] Loading core plugin: ml2
  2016-05-16 12:39:07.960 13463 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['flat', 'gre', 'vlan']
  2016-05-16 12:39:07.963 13463 INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['external']
  2016-05-16 12:39:07.966 13463 INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: {'vlan': [(300, 4000)]}
  2016-05-16 12:39:07.972 13463 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'vlan', 'gre']
  2016-05-16 12:39:07.973 13463 INFO neutron.plugins.ml2.managers [-] Registered types: ['flat', 'vlan', 'gre']
  2016-05-16 12:39:07.973 13463 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['gre']
  2016-05-16 12:39:07.974 13463 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: []
  2016-05-16 12:39:07.974 13463 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: []
  2016-05-16 12:39:07.975 13463 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: []
  2016-05-16 12:39:07.975 13463 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['openvswitch']
  2016-05-16 12:39:07.977 13463 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['openvswitch']
  2016-05-16 12:39:07.978 13463 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['openvswitch']
  2016-05-16 12:39:08.062 13463 DEBUG neutron.callbacks.manager [-] Subscribe: <bound method Ml2Plugin.validate_network_rbac_policy_change of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy before_create subscribe /usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
  2016-05-16 12:39:08.062 13463 DEBUG neutron.callbacks.manager [-] Subscribe: <bound method Ml2Plugin.validate_network_rbac_policy_change of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy before_update subscribe /usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
  2016-05-16 12:39:08.063 13463 DEBUG neutron.callbacks.manager [-] Subscribe: <bound method Ml2Plugin.validate_network_rbac_policy_change of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy before_delete subscribe /usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
  2016-05-16 12:39:08.064 13463 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat'
  2016-05-16 12:39:08.064 13463 INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete
  2016-05-16 12:39:08.065 13463 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'vlan'
  2016-05-16 12:39:08.115 13463 DEBUG oslo_db.sqlalchemy.engines [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
  }}}

  We are able to use the following to populate a fresh database, then
  neutron will start.

  {{{
  su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron
  }}}

  However as you can imagine its quite important that we are able to
  upgrade the current database!

  Appreciate any help, or links to previous related bugs that may
  improve our situation.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1582224/+subscriptions
Follow ups

[Bug 1582224] Re: Post liberty upgrade neutron-server wont start - seemingly related to ml2_vlan_allocations table
From: Launchpad Bug Tracker, 2016-07-19