← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #53021

[Bug 1597817] [NEW] Connection to neutron failed: HTTPConnectionPool read timed out

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
During 9.0RC2 Certification Testing Sutie run, the following error log appeared when
executing "NeutronNetworks.create_and_delete_routers" task:

2016-06-27 04:58:07.004 21234 DEBUG neutronclient.client [-] throwing ConnectionFailed : HTTPConnectionPool(host='172.16.184.3', port=9696): Read timed out. (read timeout=180.0) _cs_request /opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py:105
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner [-] Connection to neutron failed: HTTPConnectionPool(host='172.16.184.3', port=9696): Read timed out. (read timeout=180.0)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner Traceback (most recent call last):
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 66, in _run_scenario_once
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     deprecated_output = getattr(scenario_inst, method_name)(**kwargs)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/neutron/network.py", line 239, in create_and_delete_routers
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     self._remove_interface_router(subnet["subnet"], router["router"])
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     f = func(self, *args, **kwargs)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/neutron/utils.py", line 373, in _remove_interface_router
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     router["id"], {"subnet_id": subnet["id"]})
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 752, in remove_interface_router
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     "/remove_router_interface", body=body)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 351, in put
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     headers=headers, params=params)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 319, in retry_request
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     headers=headers, params=params)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in do_request
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     resp, replybody = self.httpclient.do_request(action, method, body=body)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py", line 170, in do_request
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     **kwargs)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py", line 106, in _cs_request
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     raise exceptions.ConnectionFailed(reason=e)
2016-06-27 04:58:07.009 21234 ERROR rally.task.runner ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='172.16.184.3', port=9696): Read timed out. (read timeout=180.0)

Steps to reproduce
==================
Install Rally on masternode and run task "NeutronNetworks.create_and_delete_routers":

1. cd /opt/stack
2. source .venv/bin/activate
3. rally task start /opt/stack/rally-scenarios/neutron/create_and_delete_routers.yaml

Expected result
===============
Task run without errors

Actual result
=============
Errors described above in logs

Environment
===========
Scale 200-node lab (ENV-10):
  3 controllers
  199 compute nodes
  20 ceph nodes
  DVR enabled
  XVLAN
Image RC2 build:
  fuel-9.0-mos-495-2016-06-16_18-18-00.iso

Diagnostic snapshot:
  http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz

** Affects: neutron
     Importance: Undecided
         Status: New

** Attachment added: "Logs and YAML Rally scenario file"
   https://bugs.launchpad.net/bugs/1597817/+attachment/4692911/+files/attachment.tar.gz

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1597817

Title:
  Connection to neutron failed: HTTPConnectionPool read timed out

Status in neutron:
  New

Bug description:
  Description
  ===========
  During 9.0RC2 Certification Testing Sutie run, the following error log appeared when
  executing "NeutronNetworks.create_and_delete_routers" task:

  2016-06-27 04:58:07.004 21234 DEBUG neutronclient.client [-] throwing ConnectionFailed : HTTPConnectionPool(host='172.16.184.3', port=9696): Read timed out. (read timeout=180.0) _cs_request /opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py:105
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner [-] Connection to neutron failed: HTTPConnectionPool(host='172.16.184.3', port=9696): Read timed out. (read timeout=180.0)
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner Traceback (most recent call last):
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 66, in _run_scenario_once
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     deprecated_output = getattr(scenario_inst, method_name)(**kwargs)
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/neutron/network.py", line 239, in create_and_delete_routers
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     self._remove_interface_router(subnet["subnet"], router["router"])
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     f = func(self, *args, **kwargs)
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/neutron/utils.py", line 373, in _remove_interface_router
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     router["id"], {"subnet_id": subnet["id"]})
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 752, in remove_interface_router
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     "/remove_router_interface", body=body)
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 351, in put
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     headers=headers, params=params)
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 319, in retry_request
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     headers=headers, params=params)
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in do_request
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     resp, replybody = self.httpclient.do_request(action, method, body=body)
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py", line 170, in do_request
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     **kwargs)
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py", line 106, in _cs_request
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner     raise exceptions.ConnectionFailed(reason=e)
  2016-06-27 04:58:07.009 21234 ERROR rally.task.runner ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='172.16.184.3', port=9696): Read timed out. (read timeout=180.0)

  Steps to reproduce
  ==================
  Install Rally on masternode and run task "NeutronNetworks.create_and_delete_routers":

  1. cd /opt/stack
  2. source .venv/bin/activate
  3. rally task start /opt/stack/rally-scenarios/neutron/create_and_delete_routers.yaml

  Expected result
  ===============
  Task run without errors

  Actual result
  =============
  Errors described above in logs

  Environment
  ===========
  Scale 200-node lab (ENV-10):
    3 controllers
    199 compute nodes
    20 ceph nodes
    DVR enabled
    XVLAN
  Image RC2 build:
    fuel-9.0-mos-495-2016-06-16_18-18-00.iso

  Diagnostic snapshot:
    http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1597817/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next