yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1913451] [NEW] nova-lvm

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lee Yarwood <1913451@xxxxxxxxxxxxxxxxxx>
Date: Wed, 27 Jan 2021 14:52:57 -0000
Reply-to: Bug 1913451 <1913451@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========

nova-lvm is often failing due to failures creating snapshots of
instances during tempest runs:

https://zuul.opendev.org/t/openstack/builds?job_name=nova-
lvm&branch=master

I *think* swift that is being used by g-api as the backing store is
running out of space due to the nova-lvm job creating two file backed
24G volume groups, one for n-cpu and another for c-vol:

https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-n-cpu.txt#37491

Jan 27 11:34:04.214271 ubuntu-focal-rax-ord-0022741499 nova-
compute[90693]: ERROR oslo_messaging.rpc.server
glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway: The
proxy server received an invalid: response from an upstream server.:
Apache/2.4.41 (Ubuntu) Server at 10.209.161.68 Port 80

https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-g-api.txt

Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi [None req-a3f9e82f-5a4c-40fa-a102-7a1a3a7998a6 tempest-ServerActionsTestJSON-1412104246 tempest-ServerActionsTestJSON-1412104246-project] Caught error: Failed to add object to Swift.
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi Traceback (most recent call last):
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/swift/store.py", line 1014, in add
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     self._delete_stale_chunks(
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     self.force_reraise()
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     raise value
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/swift/store.py", line 1003, in add
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     manager.get_connection().put_object(
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 2015, in put_object
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     return self._retry(reset_func, put_object, container, obj, contents,
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 1898, in _retry
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     reset_func(func, *args, **kwargs)
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 1995, in _default_reset
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     raise ClientException('put_object(%r, %r, ...) failure and no '
Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance', '0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005', ...) failure and no ability to reset contents for reupload.

https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-s-proxy.txt?severity=0

Jan 27 11:34:01.340337 ubuntu-focal-rax-ord-0022741499 proxy-server[81160]: ERROR with Object server 127.0.0.1:6613/sdb1 re: Trying to write to /AUTH_49fdf92e98d9449b92eddfd7528e34b8/glance/0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005: #012Traceback (most recent call last):#012  File "/opt/stack/swift/swift/proxy/controllers/obj.py", line 1678, in _send_chunk#012    self.conn.send(to_send)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/green/http/client.py", line 1049, in send#012    self.sock.sendall(data)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 405, in sendall#012    tail += self.send(data[tail:], flags)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 396, in send#012    return self._send_loop(self.fd.send, data, flags)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 383, in _send_loop#012    return send_method(data, *args)#012BrokenPipeError: [Errno 32] Broken pipe (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68)
Jan 27 11:34:01.348569 ubuntu-focal-rax-ord-0022741499 swift-proxy-server[81160]: proxy-server: ERROR with Object server 127.0.0.1:6613/sdb1 re: Trying to write to /AUTH_49fdf92e98d9449b92eddfd7528e34b8/glance/0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005: #012Traceback (most recent call last):#012  File "/opt/stack/swift/swift/proxy/controllers/obj.py", line 1678, in _send_chunk#012    self.conn.send(to_send)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/green/http/client.py", line 1049, in send#012    self.sock.sendall(data)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 405, in sendall#012    tail += self.send(data[tail:], flags)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 396, in send#012    return self._send_loop(self.fd.send, data, flags)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 383, in _send_loop#012    return send_method(data, *args)#012BrokenPipeError: [Errno 32] Broken pipe (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68)
Jan 27 11:34:01.348569 ubuntu-focal-rax-ord-0022741499 swift-proxy-server[81160]: proxy-server: Object PUT exceptions during send, 0/1 required connections (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1913451

Title:
  nova-lvm

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  nova-lvm is often failing due to failures creating snapshots of
  instances during tempest runs:

  https://zuul.opendev.org/t/openstack/builds?job_name=nova-
  lvm&branch=master

  I *think* swift that is being used by g-api as the backing store is
  running out of space due to the nova-lvm job creating two file backed
  24G volume groups, one for n-cpu and another for c-vol:

  https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-n-cpu.txt#37491

  Jan 27 11:34:04.214271 ubuntu-focal-rax-ord-0022741499 nova-
  compute[90693]: ERROR oslo_messaging.rpc.server
  glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway:
  The proxy server received an invalid: response from an upstream
  server.: Apache/2.4.41 (Ubuntu) Server at 10.209.161.68 Port 80

  https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-g-api.txt

  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi [None req-a3f9e82f-5a4c-40fa-a102-7a1a3a7998a6 tempest-ServerActionsTestJSON-1412104246 tempest-ServerActionsTestJSON-1412104246-project] Caught error: Failed to add object to Swift.
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi Traceback (most recent call last):
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/swift/store.py", line 1014, in add
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     self._delete_stale_chunks(
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     self.force_reraise()
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     raise value
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/glance_store/_drivers/swift/store.py", line 1003, in add
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     manager.get_connection().put_object(
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 2015, in put_object
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     return self._retry(reset_func, put_object, container, obj, contents,
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 1898, in _retry
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     reset_func(func, *args, **kwargs)
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi   File "/usr/local/lib/python3.8/dist-packages/swiftclient/client.py", line 1995, in _default_reset
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi     raise ClientException('put_object(%r, %r, ...) failure and no '
  Jan 27 11:34:03.231171 ubuntu-focal-rax-ord-0022741499 devstack@g-api.service[93210]: ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance', '0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005', ...) failure and no ability to reset contents for reupload.

  https://zuul.opendev.org/t/openstack/build/4f259f6d6cd444c2acea08a6e409a7f9/log/controller/logs/screen-s-proxy.txt?severity=0

  Jan 27 11:34:01.340337 ubuntu-focal-rax-ord-0022741499 proxy-server[81160]: ERROR with Object server 127.0.0.1:6613/sdb1 re: Trying to write to /AUTH_49fdf92e98d9449b92eddfd7528e34b8/glance/0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005: #012Traceback (most recent call last):#012  File "/opt/stack/swift/swift/proxy/controllers/obj.py", line 1678, in _send_chunk#012    self.conn.send(to_send)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/green/http/client.py", line 1049, in send#012    self.sock.sendall(data)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 405, in sendall#012    tail += self.send(data[tail:], flags)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 396, in send#012    return self._send_loop(self.fd.send, data, flags)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 383, in _send_loop#012    return send_method(data, *args)#012BrokenPipeError: [Errno 32] Broken pipe (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68)
  Jan 27 11:34:01.348569 ubuntu-focal-rax-ord-0022741499 swift-proxy-server[81160]: proxy-server: ERROR with Object server 127.0.0.1:6613/sdb1 re: Trying to write to /AUTH_49fdf92e98d9449b92eddfd7528e34b8/glance/0d60bd7d-7b97-4cb1-9105-19b917379ee4-00005: #012Traceback (most recent call last):#012  File "/opt/stack/swift/swift/proxy/controllers/obj.py", line 1678, in _send_chunk#012    self.conn.send(to_send)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/green/http/client.py", line 1049, in send#012    self.sock.sendall(data)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 405, in sendall#012    tail += self.send(data[tail:], flags)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 396, in send#012    return self._send_loop(self.fd.send, data, flags)#012  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 383, in _send_loop#012    return send_method(data, *args)#012BrokenPipeError: [Errno 32] Broken pipe (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68)
  Jan 27 11:34:01.348569 ubuntu-focal-rax-ord-0022741499 swift-proxy-server[81160]: proxy-server: Object PUT exceptions during send, 0/1 required connections (txn: tx7f73b39856eb45c898ce6-0060114fa5) (client_ip: 10.209.161.68)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1913451/+subscriptions
Follow ups

[Bug 1913451] Re: nova-lvm appears to be running out of swift storage space
From: Balazs Gibizer, 2021-02-04