← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1954430] [NEW] Openstack image command-line operations fail for large images in Kolla Victoria

 

Public bug reported:

**Bug Report**

What happened:

Openstack image upload and save commands fail with the following
'Connection reset by peer' error, for only large images.

# openstack image save windows-server2016 --file /tmp/t --debug

GET call to image for http://172.16.201.254:9292/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file used request id req-d15c2664-b776-4486-8f12-fb0eb2bdfef8
Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 438, in _error_catcher
    yield
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/lib64/python3.6/http/client.py", line 463, in read
    n = self.readinto(b)
  File "/usr/lib64/python3.6/http/client.py", line 507, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib64/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 758, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 455, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
    chunk_size=chunk_size):
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 761, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
    return super(Command, self).run(parsed_args)
  File "/usr/local/lib/python3.6/site-packages/cliff/command.py", line 186, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/local/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 847, in take_action
    image_client.download_image(image.id, stream=True, output=output_file)
  File "/usr/local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
    self, stream=stream, output=output, chunk_size=chunk_size)
  File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
    "Unable to download image: %s" % e)
openstack.exceptions.SDKException: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
clean_up SaveImage: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
END return value: 1


The haproxy logs show a 'CD--' status, which suggest that the client
aborted the connection.

Dec 10 11:46:58 node003 haproxy[43]: 172.16.201.1:51626
[10/Dec/2021:11:46:47.358] glance_api_front glance_api_back/node001
0/0/0/219/11319 200 1132157897 - - CD-- 297/3/0/0/0 0/0 "GET
/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file HTTP/1.1"


The glance-api server logs  the following 'Connection reset by peer'
error, same as openstack cmdline client.

node001: 2021-12-10 11:52:15.247 58 INFO eventlet.wsgi.server [req-4af3af95-6363-4857-ad82-7a30c784bc40 a8814484a71c4c5193282bcfd5e6ff3f fea7b584dcc343c19c65498e099ffd80 - default default] Traceback (most recent call last):
node001:   File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 591, in handle_one_response
node001:     write(b''.join(towrite))
node001:   File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 537, in write
node001:     wfile.writelines(towrite)
node001:   File "/usr/lib64/python3.6/socket.py", line 604, in write
node001:     return self._sock.send(b)
node001:   File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 397, in send
node001:     return self._send_loop(self.fd.send, data, flags)
node001:   File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
node001:     return send_method(data, *args)
node001: ConnectionResetError: [Errno 104] Connection reset by peer


Can this issue be resolved by modifying the default haproxy
configuration?


What you expected to happen:

The 'openstack image create' and 'openstack image save' commands should
execute succesfully.

How to reproduce it (minimal and precise):

Try to upload a 40GB+ image with 'openstack image create' command, in a
Kolla Victoria setup, in High availability mode, with Haproxy and
virtual IPs.

**Environment**:

* OS : CentOS Linux 8
* Kernel : 4.18.0-305.3.1.el8.x86_64
* Docker version : 20.10.8(Docker Engine - Community)
* Kolla-ansible version : 11.1.0
* Openstack version : Victoria

** Affects: glance
     Importance: Undecided
         Status: New

** Affects: kolla
     Importance: Undecided
         Status: New


** Tags: client command-line glance haproxy kolla network

** Also affects: glance
   Importance: Undecided
       Status: New

** Description changed:

- 
  **Bug Report**
  
  What happened:
  
  Openstack image upload and save commands fail with the following
  'Connection reset by peer' error, for only large images.
  
  # openstack image save windows-server2016 --file /tmp/t --debug
  
  GET call to image for http://172.16.201.254:9292/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file used request id req-d15c2664-b776-4486-8f12-fb0eb2bdfef8
  Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  Traceback (most recent call last):
-   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 438, in _error_catcher
-     yield
-   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 519, in read
-     data = self._fp.read(amt) if not fp_closed else b""
-   File "/usr/lib64/python3.6/http/client.py", line 463, in read
-     n = self.readinto(b)
-   File "/usr/lib64/python3.6/http/client.py", line 507, in readinto
-     n = self.fp.readinto(b)
-   File "/usr/lib64/python3.6/socket.py", line 586, in readinto
-     return self._sock.recv_into(b)
+   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 438, in _error_catcher
+     yield
+   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 519, in read
+     data = self._fp.read(amt) if not fp_closed else b""
+   File "/usr/lib64/python3.6/http/client.py", line 463, in read
+     n = self.readinto(b)
+   File "/usr/lib64/python3.6/http/client.py", line 507, in readinto
+     n = self.fp.readinto(b)
+   File "/usr/lib64/python3.6/socket.py", line 586, in readinto
+     return self._sock.recv_into(b)
  ConnectionResetError: [Errno 104] Connection reset by peer
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
-   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 758, in generate
-     for chunk in self.raw.stream(chunk_size, decode_content=True):
-   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 576, in stream
-     data = self.read(amt=amt, decode_content=decode_content)
-   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 541, in read
-     raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
-   File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
-     self.gen.throw(type, value, traceback)
-   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 455, in _error_catcher
-     raise ProtocolError("Connection broken: %r" % e, e)
+   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 758, in generate
+     for chunk in self.raw.stream(chunk_size, decode_content=True):
+   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 576, in stream
+     data = self.read(amt=amt, decode_content=decode_content)
+   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 541, in read
+     raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
+   File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
+     self.gen.throw(type, value, traceback)
+   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 455, in _error_catcher
+     raise ProtocolError("Connection broken: %r" % e, e)
  urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
-   File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
-     chunk_size=chunk_size):
-   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 761, in generate
-     raise ChunkedEncodingError(e)
+   File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
+     chunk_size=chunk_size):
+   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 761, in generate
+     raise ChunkedEncodingError(e)
  requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
-   File "/usr/local/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
-     result = cmd.run(parsed_args)
-   File "/usr/local/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
-     return super(Command, self).run(parsed_args)
-   File "/usr/local/lib/python3.6/site-packages/cliff/command.py", line 186, in run
-     return_code = self.take_action(parsed_args) or 0
-   File "/usr/local/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 847, in take_action
-     image_client.download_image(image.id, stream=True, output=output_file)
-   File "/usr/local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
-     self, stream=stream, output=output, chunk_size=chunk_size)
-   File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
-     "Unable to download image: %s" % e)
+   File "/usr/local/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
+     result = cmd.run(parsed_args)
+   File "/usr/local/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
+     return super(Command, self).run(parsed_args)
+   File "/usr/local/lib/python3.6/site-packages/cliff/command.py", line 186, in run
+     return_code = self.take_action(parsed_args) or 0
+   File "/usr/local/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 847, in take_action
+     image_client.download_image(image.id, stream=True, output=output_file)
+   File "/usr/local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
+     self, stream=stream, output=output, chunk_size=chunk_size)
+   File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
+     "Unable to download image: %s" % e)
  openstack.exceptions.SDKException: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  clean_up SaveImage: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  END return value: 1
  
  
  The haproxy logs show a 'CD--' status, which suggest that the client
  aborted the connection.
  
- 
- Dec 10 11:46:58 node003 haproxy[43]: 172.16.201.1:51626 [10/Dec/2021:11:46:47.358] glance_api_front glance_api_back/node001 0/0/0/219/11319 200 1132157897 - - CD-- 297/3/0/0/0 0/0 "GET /v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file HTTP/1.1"
+ Dec 10 11:46:58 node003 haproxy[43]: 172.16.201.1:51626
+ [10/Dec/2021:11:46:47.358] glance_api_front glance_api_back/node001
+ 0/0/0/219/11319 200 1132157897 - - CD-- 297/3/0/0/0 0/0 "GET
+ /v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file HTTP/1.1"
  
  
- The glance-api server logs  the following 'Connection reset by peer' error, same as openstack cmdline client.
+ The glance-api server logs  the following 'Connection reset by peer'
+ error, same as openstack cmdline client.
  
  node001: 2021-12-10 11:52:15.247 58 INFO eventlet.wsgi.server [req-4af3af95-6363-4857-ad82-7a30c784bc40 a8814484a71c4c5193282bcfd5e6ff3f fea7b584dcc343c19c65498e099ffd80 - default default] Traceback (most recent call last):
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 591, in handle_one_response
  node001:     write(b''.join(towrite))
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 537, in write
  node001:     wfile.writelines(towrite)
  node001:   File "/usr/lib64/python3.6/socket.py", line 604, in write
  node001:     return self._sock.send(b)
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 397, in send
  node001:     return self._send_loop(self.fd.send, data, flags)
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
  node001:     return send_method(data, *args)
  node001: ConnectionResetError: [Errno 104] Connection reset by peer
+ 
  
  
  Can this issue be resolved by modifying the default haproxy configuration?
  
  
  What you expected to happen:
  
  The 'openstack image create' and 'openstack image save' commands should
  execute succesfully.
  
  How to reproduce it (minimal and precise):
  
  Try to upload a 40GB+ image with 'openstack image create' command, in a
  Kolla Victoria setup, in High availability mode, with Haproxy and
  virtual IPs.
  
  **Environment**:
  
  * OS : CentOS Linux 8
  * Kernel : 4.18.0-305.3.1.el8.x86_64
  * Docker version : 20.10.8(Docker Engine - Community)
  * Kolla-ansible version : 11.1.0
  * Openstack version : Victoria

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1954430

Title:
  Openstack image command-line operations fail for large images in Kolla
  Victoria

Status in Glance:
  New
Status in kolla:
  New

Bug description:
  **Bug Report**

  What happened:

  Openstack image upload and save commands fail with the following
  'Connection reset by peer' error, for only large images.

  # openstack image save windows-server2016 --file /tmp/t --debug

  GET call to image for http://172.16.201.254:9292/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file used request id req-d15c2664-b776-4486-8f12-fb0eb2bdfef8
  Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 438, in _error_catcher
      yield
    File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 519, in read
      data = self._fp.read(amt) if not fp_closed else b""
    File "/usr/lib64/python3.6/http/client.py", line 463, in read
      n = self.readinto(b)
    File "/usr/lib64/python3.6/http/client.py", line 507, in readinto
      n = self.fp.readinto(b)
    File "/usr/lib64/python3.6/socket.py", line 586, in readinto
      return self._sock.recv_into(b)
  ConnectionResetError: [Errno 104] Connection reset by peer

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 758, in generate
      for chunk in self.raw.stream(chunk_size, decode_content=True):
    File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 576, in stream
      data = self.read(amt=amt, decode_content=decode_content)
    File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 541, in read
      raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
    File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
      self.gen.throw(type, value, traceback)
    File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 455, in _error_catcher
      raise ProtocolError("Connection broken: %r" % e, e)
  urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
      chunk_size=chunk_size):
    File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 761, in generate
      raise ChunkedEncodingError(e)
  requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
      result = cmd.run(parsed_args)
    File "/usr/local/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
      return super(Command, self).run(parsed_args)
    File "/usr/local/lib/python3.6/site-packages/cliff/command.py", line 186, in run
      return_code = self.take_action(parsed_args) or 0
    File "/usr/local/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 847, in take_action
      image_client.download_image(image.id, stream=True, output=output_file)
    File "/usr/local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
      self, stream=stream, output=output, chunk_size=chunk_size)
    File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
      "Unable to download image: %s" % e)
  openstack.exceptions.SDKException: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  clean_up SaveImage: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  END return value: 1


  The haproxy logs show a 'CD--' status, which suggest that the client
  aborted the connection.

  Dec 10 11:46:58 node003 haproxy[43]: 172.16.201.1:51626
  [10/Dec/2021:11:46:47.358] glance_api_front glance_api_back/node001
  0/0/0/219/11319 200 1132157897 - - CD-- 297/3/0/0/0 0/0 "GET
  /v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file HTTP/1.1"


  The glance-api server logs  the following 'Connection reset by peer'
  error, same as openstack cmdline client.

  node001: 2021-12-10 11:52:15.247 58 INFO eventlet.wsgi.server [req-4af3af95-6363-4857-ad82-7a30c784bc40 a8814484a71c4c5193282bcfd5e6ff3f fea7b584dcc343c19c65498e099ffd80 - default default] Traceback (most recent call last):
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 591, in handle_one_response
  node001:     write(b''.join(towrite))
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 537, in write
  node001:     wfile.writelines(towrite)
  node001:   File "/usr/lib64/python3.6/socket.py", line 604, in write
  node001:     return self._sock.send(b)
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 397, in send
  node001:     return self._send_loop(self.fd.send, data, flags)
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
  node001:     return send_method(data, *args)
  node001: ConnectionResetError: [Errno 104] Connection reset by peer


  
  Can this issue be resolved by modifying the default haproxy configuration?


  What you expected to happen:

  The 'openstack image create' and 'openstack image save' commands
  should execute succesfully.

  How to reproduce it (minimal and precise):

  Try to upload a 40GB+ image with 'openstack image create' command, in
  a Kolla Victoria setup, in High availability mode, with Haproxy and
  virtual IPs.

  **Environment**:

  * OS : CentOS Linux 8
  * Kernel : 4.18.0-305.3.1.el8.x86_64
  * Docker version : 20.10.8(Docker Engine - Community)
  * Kolla-ansible version : 11.1.0
  * Openstack version : Victoria

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1954430/+subscriptions



Follow ups