yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

There are DBDeadlock errors in the neutron server logs:

http://logs.openstack.org/40/122240/19/gate/gate-tempest-dsvm-neutron-
full/4ef0a02/logs/screen-q-svc.txt.gz?level=TRACE#_2015-02-27_18_05_43_389

** Also affects: neutron
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1426543

Title:
  Spike in DBDeadlock errors in update_floatingip_statuses since 2/27

Status in OpenStack Neutron (virtual network service):
  New
Status in OpenStack Compute (Nova):
  New

Bug description:
  http://logs.openstack.org/40/122240/19/gate/gate-tempest-dsvm-neutron-
  full/4ef0a02/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-02-27_18_05_22_444

  2015-02-27 18:05:22.444 8433 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager Traceback (most recent call last):
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/opt/stack/new/nova/nova/compute/manager.py", line 1684, in _allocate_network_async
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     dhcp_options=dhcp_options)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 395, in allocate_for_instance
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     net_ids, neutron=neutron)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 226, in _get_available_networks
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     nets = neutron.list_networks(**search_opts).get('networks', [])
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     ret = self.function(instance, *args, **kwargs)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 524, in list_networks
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     **_params)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 304, in list
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     for r in self._pagination(collection, path, **params):
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 317, in _pagination
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     res = self.get(path, params=params)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 290, in get
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     headers=headers, params=params)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 267, in retry_request
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     headers=headers, params=params)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 197, in do_request
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     content_type=self.content_type())
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 172, in do_request
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     **kwargs)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 108, in _cs_request
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     raise exceptions.ConnectionFailed(reason=e)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='127.0.0.1', port=9696): Max retries exceeded with url: /v2.0/networks.json?tenant_id=1e707ab2d2be40a1902f3352c91a615a&shared=False (Caused by ReadTimeoutError("HTTPConnectionPool(host='127.0.0.1', port=9696): Read timed out. (read timeout=30)",))
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager 


  http://goo.gl/UMI2vZ

  120 hits in the last 24 hours.  There aren't any new python-
  neutronclient releases in that time so that's not it and there are no
  changes to nova.network.neutronv2 in the last 24 hours, so have to
  look at recent compute manager changes.

  There were a few releases of the requests library this week but those
  were on 2/23 and 2/24 so if that was it we should have seen this by
  now.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1426543/+subscriptions