← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #28891

[Bug 1426543] [NEW] Spike in neutronclient connection failures due to read timeouts since 2/27

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

http://logs.openstack.org/40/122240/19/gate/gate-tempest-dsvm-neutron-
full/4ef0a02/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-02-27_18_05_22_444

2015-02-27 18:05:22.444 8433 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager Traceback (most recent call last):
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/opt/stack/new/nova/nova/compute/manager.py", line 1684, in _allocate_network_async
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     dhcp_options=dhcp_options)
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 395, in allocate_for_instance
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     net_ids, neutron=neutron)
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 226, in _get_available_networks
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     nets = neutron.list_networks(**search_opts).get('networks', [])
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     ret = self.function(instance, *args, **kwargs)
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 524, in list_networks
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     **_params)
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 304, in list
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     for r in self._pagination(collection, path, **params):
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 317, in _pagination
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     res = self.get(path, params=params)
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 290, in get
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     headers=headers, params=params)
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 267, in retry_request
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     headers=headers, params=params)
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 197, in do_request
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     content_type=self.content_type())
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 172, in do_request
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     **kwargs)
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 108, in _cs_request
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     raise exceptions.ConnectionFailed(reason=e)
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='127.0.0.1', port=9696): Max retries exceeded with url: /v2.0/networks.json?tenant_id=1e707ab2d2be40a1902f3352c91a615a&shared=False (Caused by ReadTimeoutError("HTTPConnectionPool(host='127.0.0.1', port=9696): Read timed out. (read timeout=30)",))
2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager 


http://goo.gl/UMI2vZ

120 hits in the last 24 hours.  There aren't any new python-
neutronclient releases in that time so that's not it and there are no
changes to nova.network.neutronv2 in the last 24 hours, so have to look
at recent compute manager changes.

There were a few releases of the requests library this week but those
were on 2/23 and 2/24 so if that was it we should have seen this by now.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: network neutron

** Tags added: network

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1426543

Title:
  Spike in neutronclient connection failures due to read timeouts since
  2/27

Status in OpenStack Compute (Nova):
  New

Bug description:
  http://logs.openstack.org/40/122240/19/gate/gate-tempest-dsvm-neutron-
  full/4ef0a02/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-02-27_18_05_22_444

  2015-02-27 18:05:22.444 8433 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager Traceback (most recent call last):
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/opt/stack/new/nova/nova/compute/manager.py", line 1684, in _allocate_network_async
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     dhcp_options=dhcp_options)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 395, in allocate_for_instance
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     net_ids, neutron=neutron)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 226, in _get_available_networks
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     nets = neutron.list_networks(**search_opts).get('networks', [])
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     ret = self.function(instance, *args, **kwargs)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 524, in list_networks
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     **_params)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 304, in list
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     for r in self._pagination(collection, path, **params):
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 317, in _pagination
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     res = self.get(path, params=params)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 290, in get
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     headers=headers, params=params)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 267, in retry_request
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     headers=headers, params=params)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 197, in do_request
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     content_type=self.content_type())
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 172, in do_request
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     **kwargs)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 108, in _cs_request
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager     raise exceptions.ConnectionFailed(reason=e)
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='127.0.0.1', port=9696): Max retries exceeded with url: /v2.0/networks.json?tenant_id=1e707ab2d2be40a1902f3352c91a615a&shared=False (Caused by ReadTimeoutError("HTTPConnectionPool(host='127.0.0.1', port=9696): Read timed out. (read timeout=30)",))
  2015-02-27 18:05:22.444 8433 TRACE nova.compute.manager 


  http://goo.gl/UMI2vZ

  120 hits in the last 24 hours.  There aren't any new python-
  neutronclient releases in that time so that's not it and there are no
  changes to nova.network.neutronv2 in the last 24 hours, so have to
  look at recent compute manager changes.

  There were a few releases of the requests library this week but those
  were on 2/23 and 2/24 so if that was it we should have seen this by
  now.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1426543/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next