yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1808063] Re: glanceclient.exc.HTTPBadGateway: 502 Proxy Error during server snapshot

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 13 Dec 2018 16:41:54 -0000
Reply-to: Bug 1808063 <1808063@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
There is definitely some kind of infinite loop here:

Dec 13 14:43:21.412172 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]: /opt/stack/new/glance/glance/quota/__init__.py:168: ResourceWarning: unclosed <socket.socket fd=14, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('104.130.211.192', 52252), raddr=('104.130.211.192', 8080)>
Dec 13 14:43:21.412563 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   return getattr(self.tags, name)
Dec 13 14:43:21.413081 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]: Fatal Python error: Cannot recover from stack overflow.
Dec 13 14:43:21.413440 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]: Thread 0x00007f4ff7ac8700 (most recent call first):
Dec 13 14:43:21.413817 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/lib/python3.5/threading.py", line 297 in wait
Dec 13 14:43:21.414179 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/lib/python3.5/threading.py", line 549 in wait
Dec 13 14:43:21.414511 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 926 in _heartbeat_thread_job
Dec 13 14:43:21.414869 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/lib/python3.5/threading.py", line 862 in run
Dec 13 14:43:21.415584 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
Dec 13 14:43:21.415951 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap
Dec 13 14:43:21.416261 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]: Thread 0x00007f4ff8349700 (most recent call first):
Dec 13 14:43:21.416622 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/lib/python3.5/threading.py", line 297 in wait
Dec 13 14:43:21.416995 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/lib/python3.5/threading.py", line 549 in wait
Dec 13 14:43:21.417357 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 926 in _heartbeat_thread_job
Dec 13 14:43:21.417721 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/lib/python3.5/threading.py", line 862 in run
Dec 13 14:43:21.418085 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
Dec 13 14:43:21.418450 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap
Dec 13 14:43:21.419113 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]: Current thread 0x00007f50070af780 (most recent call first):
Dec 13 14:43:21.419499 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__
Dec 13 14:43:21.419875 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__
Dec 13 14:43:21.421218 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__
Dec 13 14:43:21.421577 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__
Dec 13 14:43:21.421947 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__
Dec 13 14:43:21.422307 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__
Dec 13 14:43:21.422671 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]:   File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__

** Also affects: glance
   Importance: Undecided
       Status: New

** Changed in: glance
   Importance: Undecided => High

** Changed in: glance
       Status: New => Confirmed

** Summary changed:

- glanceclient.exc.HTTPBadGateway: 502 Proxy Error during server snapshot
+ glanceclient.exc.HTTPBadGateway: 502 Proxy Error during server snapshot - infinite loop in File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1808063

Title:
  glanceclient.exc.HTTPBadGateway: 502 Proxy Error during server
  snapshot - infinite loop in File
  "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in
  __getattr__

Status in Glance:
  Confirmed
Status in OpenStack-Gate:
  Confirmed

Bug description:
  Seen here:

  http://logs.openstack.org/56/621856/10/check/tempest-full-
  py3/85b3f05/controller/logs/screen-n-api.txt.gz#_Dec_11_19_57_35_024101

  Dec 11 19:57:35.024101 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi [None req-76b7b44d-b425-46f0-84c9-307f3fcf9795 tempest-ImagesOneServerNegativeTestJSON-1724249280 tempest-ImagesOneServerNegativeTestJSON-1724249280] Unexpected exception in API method: glanceclient.exc.HTTPBadGateway: 502 Proxy Error: Proxy Error: The proxy server received an invalid: response from an upstream server.: The proxy server could not handle the request POST&nbsp;/image/v2/images.: Reason: Error reading from remote server: Apache/2.4.18 (Ubuntu) Server at 10.209.33.36 Port 80 (HTTP 502)
  Dec 11 19:57:35.024709 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  Dec 11 19:57:35.024947 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 655, in wrapped
  Dec 11 19:57:35.025231 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  Dec 11 19:57:35.025994 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/api/validation/__init__.py", line 110, in wrapper
  Dec 11 19:57:35.026254 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  Dec 11 19:57:35.026499 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/api/validation/__init__.py", line 110, in wrapper
  Dec 11 19:57:35.026754 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  Dec 11 19:57:35.027784 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 1059, in _action_create_image
  Dec 11 19:57:35.028062 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     extra_properties=metadata)
  Dec 11 19:57:35.028299 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 213, in _wrapped
  Dec 11 19:57:35.028530 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     return fn(self, context, instance, *args, **kwargs)
  Dec 11 19:57:35.028855 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 153, in inner
  Dec 11 19:57:35.029094 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
  Dec 11 19:57:35.029366 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 2959, in snapshot
  Dec 11 19:57:35.029754 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     extra_properties=extra_properties)
  Dec 11 19:57:35.029995 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 3032, in _create_image
  Dec 11 19:57:35.030244 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     return self.image_api.create(context, image_meta)
  Dec 11 19:57:35.030485 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/image/api.py", line 118, in create
  Dec 11 19:57:35.030716 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     return session.create(context, image_info, data=data)
  Dec 11 19:57:35.030966 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/image/glance.py", line 477, in create
  Dec 11 19:57:35.031200 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     _reraise_translated_exception()
  Dec 11 19:57:35.031546 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/image/glance.py", line 915, in _reraise_translated_exception
  Dec 11 19:57:35.031899 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     six.reraise(type(new_exc), new_exc, exc_trace)
  Dec 11 19:57:35.032269 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
  Dec 11 19:57:35.032612 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     raise value
  Dec 11 19:57:35.032977 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/image/glance.py", line 475, in create
  Dec 11 19:57:35.033337 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     data, force_activate)
  Dec 11 19:57:35.033727 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/image/glance.py", line 552, in _create_v2
  Dec 11 19:57:35.034080 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     context, 2, 'create', kwargs=sent_service_image_meta)
  Dec 11 19:57:35.034424 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/image/glance.py", line 193, in call
  Dec 11 19:57:35.034876 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     result = getattr(controller, method)(*args, **kwargs)
  Dec 11 19:57:35.035320 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.5/dist-packages/glanceclient/common/utils.py", line 598, in inner
  Dec 11 19:57:35.035693 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     return RequestIdProxy(wrapped(*args, **kwargs))
  Dec 11 19:57:35.039116 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.5/dist-packages/glanceclient/v2/images.py", line 361, in create
  Dec 11 19:57:35.039411 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     resp, body = self.http_client.post(url, headers=headers, data=image)
  Dec 11 19:57:35.039662 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.5/dist-packages/keystoneauth1/adapter.py", line 357, in post
  Dec 11 19:57:35.039899 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     return self.request(url, 'POST', **kwargs)
  Dec 11 19:57:35.040164 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.5/dist-packages/glanceclient/common/http.py", line 377, in request
  Dec 11 19:57:35.040400 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     return self._handle_response(resp)
  Dec 11 19:57:35.040632 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.5/dist-packages/glanceclient/common/http.py", line 126, in _handle_response
  Dec 11 19:57:35.040876 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi     raise exc.from_response(resp, resp.content)
  Dec 11 19:57:35.041177 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi glanceclient.exc.HTTPBadGateway: 502 Proxy Error: Proxy Error: The proxy server received an invalid: response from an upstream server.: The proxy server could not handle the request POST&nbsp;/image/v2/images.: Reason: Error reading from remote server: Apache/2.4.18 (Ubuntu) Server at 10.209.33.36 Port 80 (HTTP 502)
  Dec 11 19:57:35.041412 ubuntu-xenial-rax-dfw-0001132076 devstack@n-api.service[13547]: ERROR nova.api.openstack.wsgi 

  This bug is for e-r tracking.

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Unexpected%20exception%20in%20API%20method%3A%20glanceclient.exc.HTTPBadGateway%3A%20502%20Proxy%20Error%3A%20Proxy%20Error%3A%20The%20proxy%20server%20received%20an%20invalid%3A%20response%20from%20an%20upstream%20server.%3A%20The%20proxy%20server%20could%20not%20handle%20the%20request%20POST%26nbsp%3B%2Fimage%2Fv2%2Fimages.%3A%20Reason%3A%20Error%20reading%20from%20remote%20server%3A%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1808063/+subscriptions