yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Public bug reported:

We see this about 11 times in 7 days in the check and gate queues on the
nova functional jobs:

http://logs.openstack.org/20/459420/1/check/gate-nova-tox-functional-
ubuntu-xenial/c0bdfa6/console.html#_2017-04-25_13_55_17_705409

2017-04-25 13:55:17.704694 | Captured traceback:
2017-04-25 13:55:17.704705 | ~~~~~~~~~~~~~~~~~~~
2017-04-25 13:55:17.704720 |     Traceback (most recent call last):
2017-04-25 13:55:17.704749 |       File "nova/tests/functional/api_sample_tests/test_fixed_ips.py", line 100, in test_get_fixed_ip
2017-04-25 13:55:17.704763 |         self._test_get_fixed_ip()
2017-04-25 13:55:17.704791 |       File "nova/tests/functional/api_sample_tests/test_fixed_ips.py", line 91, in _test_get_fixed_ip
2017-04-25 13:55:17.704817 |         response = self._do_get('os-fixed-ips/192.168.1.1')
2017-04-25 13:55:17.704844 |       File "nova/tests/functional/api_samples_test_base.py", line 488, in _do_get
2017-04-25 13:55:17.704857 |         headers=headers)
2017-04-25 13:55:17.704883 |       File "nova/tests/functional/api_samples_test_base.py", line 479, in _get_response
2017-04-25 13:55:17.704901 |         headers=headers, strip_version=strip_version)
2017-04-25 13:55:17.704923 |       File "nova/tests/functional/api/client.py", line 164, in api_request
2017-04-25 13:55:17.704938 |         auth_result = self._authenticate()
2017-04-25 13:55:17.704961 |       File "nova/tests/functional/api/client.py", line 150, in _authenticate
2017-04-25 13:55:17.704972 |         headers=headers)
2017-04-25 13:55:17.704994 |       File "nova/tests/functional/api/client.py", line 138, in request
2017-04-25 13:55:17.705016 |         response = requests.request(method, url, data=body, headers=_headers)
2017-04-25 13:55:17.705059 |       File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
2017-04-25 13:55:17.705083 |         return session.request(method=method, url=url, **kwargs)
2017-04-25 13:55:17.705163 |       File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
2017-04-25 13:55:17.705202 |         resp = self.send(prep, **send_kwargs)
2017-04-25 13:55:17.705280 |       File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
2017-04-25 13:55:17.705321 |         r = adapter.send(request, **kwargs)
2017-04-25 13:55:17.705367 |       File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
2017-04-25 13:55:17.705384 |         raise ConnectionError(err, request=request)
2017-04-25 13:55:17.705409 |     requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Which API it fails on is completely random I think.

It looks like this is maybe a result of the OSAPIFixture spawning a
greenthread to run the nova-api service:

(9:19:08 AM) cdent: if for some reason the new greenthread doesn't get
yielded to when it should, the client will try to read off the socket
which nothing paying proper attention on the other side

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22requests.exceptions.ConnectionError%3A%20('Connection%20aborted.'%2C%20BadStatusLine(%5C%22''%5C%22%2C))%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22&from=7d

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: api testing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1686109

Title:
  requests.exceptions.ConnectionError: ('Connection aborted.',
  BadStatusLine("''",)) in functional tests

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  We see this about 11 times in 7 days in the check and gate queues on
  the nova functional jobs:

  http://logs.openstack.org/20/459420/1/check/gate-nova-tox-functional-
  ubuntu-xenial/c0bdfa6/console.html#_2017-04-25_13_55_17_705409

  2017-04-25 13:55:17.704694 | Captured traceback:
  2017-04-25 13:55:17.704705 | ~~~~~~~~~~~~~~~~~~~
  2017-04-25 13:55:17.704720 |     Traceback (most recent call last):
  2017-04-25 13:55:17.704749 |       File "nova/tests/functional/api_sample_tests/test_fixed_ips.py", line 100, in test_get_fixed_ip
  2017-04-25 13:55:17.704763 |         self._test_get_fixed_ip()
  2017-04-25 13:55:17.704791 |       File "nova/tests/functional/api_sample_tests/test_fixed_ips.py", line 91, in _test_get_fixed_ip
  2017-04-25 13:55:17.704817 |         response = self._do_get('os-fixed-ips/192.168.1.1')
  2017-04-25 13:55:17.704844 |       File "nova/tests/functional/api_samples_test_base.py", line 488, in _do_get
  2017-04-25 13:55:17.704857 |         headers=headers)
  2017-04-25 13:55:17.704883 |       File "nova/tests/functional/api_samples_test_base.py", line 479, in _get_response
  2017-04-25 13:55:17.704901 |         headers=headers, strip_version=strip_version)
  2017-04-25 13:55:17.704923 |       File "nova/tests/functional/api/client.py", line 164, in api_request
  2017-04-25 13:55:17.704938 |         auth_result = self._authenticate()
  2017-04-25 13:55:17.704961 |       File "nova/tests/functional/api/client.py", line 150, in _authenticate
  2017-04-25 13:55:17.704972 |         headers=headers)
  2017-04-25 13:55:17.704994 |       File "nova/tests/functional/api/client.py", line 138, in request
  2017-04-25 13:55:17.705016 |         response = requests.request(method, url, data=body, headers=_headers)
  2017-04-25 13:55:17.705059 |       File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
  2017-04-25 13:55:17.705083 |         return session.request(method=method, url=url, **kwargs)
  2017-04-25 13:55:17.705163 |       File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
  2017-04-25 13:55:17.705202 |         resp = self.send(prep, **send_kwargs)
  2017-04-25 13:55:17.705280 |       File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
  2017-04-25 13:55:17.705321 |         r = adapter.send(request, **kwargs)
  2017-04-25 13:55:17.705367 |       File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
  2017-04-25 13:55:17.705384 |         raise ConnectionError(err, request=request)
  2017-04-25 13:55:17.705409 |     requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

  Which API it fails on is completely random I think.

  It looks like this is maybe a result of the OSAPIFixture spawning a
  greenthread to run the nova-api service:

  (9:19:08 AM) cdent: if for some reason the new greenthread doesn't get
  yielded to when it should, the client will try to read off the socket
  which nothing paying proper attention on the other side

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22requests.exceptions.ConnectionError%3A%20('Connection%20aborted.'%2C%20BadStatusLine(%5C%22''%5C%22%2C))%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1686109/+subscriptions