yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #87847
[Bug 1954430] [NEW] Openstack image command-line operations fail for large images in Kolla Victoria
Public bug reported:
**Bug Report**
What happened:
Openstack image upload and save commands fail with the following
'Connection reset by peer' error, for only large images.
# openstack image save windows-server2016 --file /tmp/t --debug
GET call to image for http://172.16.201.254:9292/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file used request id req-d15c2664-b776-4486-8f12-fb0eb2bdfef8
Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 438, in _error_catcher
yield
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 519, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/usr/lib64/python3.6/http/client.py", line 463, in read
n = self.readinto(b)
File "/usr/lib64/python3.6/http/client.py", line 507, in readinto
n = self.fp.readinto(b)
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 758, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 576, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 455, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
chunk_size=chunk_size):
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 761, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/local/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
return super(Command, self).run(parsed_args)
File "/usr/local/lib/python3.6/site-packages/cliff/command.py", line 186, in run
return_code = self.take_action(parsed_args) or 0
File "/usr/local/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 847, in take_action
image_client.download_image(image.id, stream=True, output=output_file)
File "/usr/local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
self, stream=stream, output=output, chunk_size=chunk_size)
File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
"Unable to download image: %s" % e)
openstack.exceptions.SDKException: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
clean_up SaveImage: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
END return value: 1
The haproxy logs show a 'CD--' status, which suggest that the client
aborted the connection.
Dec 10 11:46:58 node003 haproxy[43]: 172.16.201.1:51626
[10/Dec/2021:11:46:47.358] glance_api_front glance_api_back/node001
0/0/0/219/11319 200 1132157897 - - CD-- 297/3/0/0/0 0/0 "GET
/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file HTTP/1.1"
The glance-api server logs the following 'Connection reset by peer'
error, same as openstack cmdline client.
node001: 2021-12-10 11:52:15.247 58 INFO eventlet.wsgi.server [req-4af3af95-6363-4857-ad82-7a30c784bc40 a8814484a71c4c5193282bcfd5e6ff3f fea7b584dcc343c19c65498e099ffd80 - default default] Traceback (most recent call last):
node001: File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 591, in handle_one_response
node001: write(b''.join(towrite))
node001: File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 537, in write
node001: wfile.writelines(towrite)
node001: File "/usr/lib64/python3.6/socket.py", line 604, in write
node001: return self._sock.send(b)
node001: File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 397, in send
node001: return self._send_loop(self.fd.send, data, flags)
node001: File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
node001: return send_method(data, *args)
node001: ConnectionResetError: [Errno 104] Connection reset by peer
Can this issue be resolved by modifying the default haproxy
configuration?
What you expected to happen:
The 'openstack image create' and 'openstack image save' commands should
execute succesfully.
How to reproduce it (minimal and precise):
Try to upload a 40GB+ image with 'openstack image create' command, in a
Kolla Victoria setup, in High availability mode, with Haproxy and
virtual IPs.
**Environment**:
* OS : CentOS Linux 8
* Kernel : 4.18.0-305.3.1.el8.x86_64
* Docker version : 20.10.8(Docker Engine - Community)
* Kolla-ansible version : 11.1.0
* Openstack version : Victoria
** Affects: glance
Importance: Undecided
Status: New
** Affects: kolla
Importance: Undecided
Status: New
** Tags: client command-line glance haproxy kolla network
** Also affects: glance
Importance: Undecided
Status: New
** Description changed:
-
**Bug Report**
What happened:
Openstack image upload and save commands fail with the following
'Connection reset by peer' error, for only large images.
# openstack image save windows-server2016 --file /tmp/t --debug
GET call to image for http://172.16.201.254:9292/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file used request id req-d15c2664-b776-4486-8f12-fb0eb2bdfef8
Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
Traceback (most recent call last):
- File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 438, in _error_catcher
- yield
- File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 519, in read
- data = self._fp.read(amt) if not fp_closed else b""
- File "/usr/lib64/python3.6/http/client.py", line 463, in read
- n = self.readinto(b)
- File "/usr/lib64/python3.6/http/client.py", line 507, in readinto
- n = self.fp.readinto(b)
- File "/usr/lib64/python3.6/socket.py", line 586, in readinto
- return self._sock.recv_into(b)
+ File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 438, in _error_catcher
+ yield
+ File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 519, in read
+ data = self._fp.read(amt) if not fp_closed else b""
+ File "/usr/lib64/python3.6/http/client.py", line 463, in read
+ n = self.readinto(b)
+ File "/usr/lib64/python3.6/http/client.py", line 507, in readinto
+ n = self.fp.readinto(b)
+ File "/usr/lib64/python3.6/socket.py", line 586, in readinto
+ return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
- File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 758, in generate
- for chunk in self.raw.stream(chunk_size, decode_content=True):
- File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 576, in stream
- data = self.read(amt=amt, decode_content=decode_content)
- File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 541, in read
- raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
- File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
- self.gen.throw(type, value, traceback)
- File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 455, in _error_catcher
- raise ProtocolError("Connection broken: %r" % e, e)
+ File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 758, in generate
+ for chunk in self.raw.stream(chunk_size, decode_content=True):
+ File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 576, in stream
+ data = self.read(amt=amt, decode_content=decode_content)
+ File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 541, in read
+ raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
+ File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
+ self.gen.throw(type, value, traceback)
+ File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 455, in _error_catcher
+ raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
- File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
- chunk_size=chunk_size):
- File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 761, in generate
- raise ChunkedEncodingError(e)
+ File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
+ chunk_size=chunk_size):
+ File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 761, in generate
+ raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
- File "/usr/local/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
- result = cmd.run(parsed_args)
- File "/usr/local/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
- return super(Command, self).run(parsed_args)
- File "/usr/local/lib/python3.6/site-packages/cliff/command.py", line 186, in run
- return_code = self.take_action(parsed_args) or 0
- File "/usr/local/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 847, in take_action
- image_client.download_image(image.id, stream=True, output=output_file)
- File "/usr/local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
- self, stream=stream, output=output, chunk_size=chunk_size)
- File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
- "Unable to download image: %s" % e)
+ File "/usr/local/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
+ result = cmd.run(parsed_args)
+ File "/usr/local/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
+ return super(Command, self).run(parsed_args)
+ File "/usr/local/lib/python3.6/site-packages/cliff/command.py", line 186, in run
+ return_code = self.take_action(parsed_args) or 0
+ File "/usr/local/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 847, in take_action
+ image_client.download_image(image.id, stream=True, output=output_file)
+ File "/usr/local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
+ self, stream=stream, output=output, chunk_size=chunk_size)
+ File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
+ "Unable to download image: %s" % e)
openstack.exceptions.SDKException: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
clean_up SaveImage: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
END return value: 1
The haproxy logs show a 'CD--' status, which suggest that the client
aborted the connection.
-
- Dec 10 11:46:58 node003 haproxy[43]: 172.16.201.1:51626 [10/Dec/2021:11:46:47.358] glance_api_front glance_api_back/node001 0/0/0/219/11319 200 1132157897 - - CD-- 297/3/0/0/0 0/0 "GET /v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file HTTP/1.1"
+ Dec 10 11:46:58 node003 haproxy[43]: 172.16.201.1:51626
+ [10/Dec/2021:11:46:47.358] glance_api_front glance_api_back/node001
+ 0/0/0/219/11319 200 1132157897 - - CD-- 297/3/0/0/0 0/0 "GET
+ /v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file HTTP/1.1"
- The glance-api server logs the following 'Connection reset by peer' error, same as openstack cmdline client.
+ The glance-api server logs the following 'Connection reset by peer'
+ error, same as openstack cmdline client.
node001: 2021-12-10 11:52:15.247 58 INFO eventlet.wsgi.server [req-4af3af95-6363-4857-ad82-7a30c784bc40 a8814484a71c4c5193282bcfd5e6ff3f fea7b584dcc343c19c65498e099ffd80 - default default] Traceback (most recent call last):
node001: File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 591, in handle_one_response
node001: write(b''.join(towrite))
node001: File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 537, in write
node001: wfile.writelines(towrite)
node001: File "/usr/lib64/python3.6/socket.py", line 604, in write
node001: return self._sock.send(b)
node001: File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 397, in send
node001: return self._send_loop(self.fd.send, data, flags)
node001: File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
node001: return send_method(data, *args)
node001: ConnectionResetError: [Errno 104] Connection reset by peer
+
Can this issue be resolved by modifying the default haproxy configuration?
What you expected to happen:
The 'openstack image create' and 'openstack image save' commands should
execute succesfully.
How to reproduce it (minimal and precise):
Try to upload a 40GB+ image with 'openstack image create' command, in a
Kolla Victoria setup, in High availability mode, with Haproxy and
virtual IPs.
**Environment**:
* OS : CentOS Linux 8
* Kernel : 4.18.0-305.3.1.el8.x86_64
* Docker version : 20.10.8(Docker Engine - Community)
* Kolla-ansible version : 11.1.0
* Openstack version : Victoria
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1954430
Title:
Openstack image command-line operations fail for large images in Kolla
Victoria
Status in Glance:
New
Status in kolla:
New
Bug description:
**Bug Report**
What happened:
Openstack image upload and save commands fail with the following
'Connection reset by peer' error, for only large images.
# openstack image save windows-server2016 --file /tmp/t --debug
GET call to image for http://172.16.201.254:9292/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file used request id req-d15c2664-b776-4486-8f12-fb0eb2bdfef8
Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 438, in _error_catcher
yield
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 519, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/usr/lib64/python3.6/http/client.py", line 463, in read
n = self.readinto(b)
File "/usr/lib64/python3.6/http/client.py", line 507, in readinto
n = self.fp.readinto(b)
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 758, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 576, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 455, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
chunk_size=chunk_size):
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 761, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/local/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
return super(Command, self).run(parsed_args)
File "/usr/local/lib/python3.6/site-packages/cliff/command.py", line 186, in run
return_code = self.take_action(parsed_args) or 0
File "/usr/local/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 847, in take_action
image_client.download_image(image.id, stream=True, output=output_file)
File "/usr/local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
self, stream=stream, output=output, chunk_size=chunk_size)
File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
"Unable to download image: %s" % e)
openstack.exceptions.SDKException: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
clean_up SaveImage: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
END return value: 1
The haproxy logs show a 'CD--' status, which suggest that the client
aborted the connection.
Dec 10 11:46:58 node003 haproxy[43]: 172.16.201.1:51626
[10/Dec/2021:11:46:47.358] glance_api_front glance_api_back/node001
0/0/0/219/11319 200 1132157897 - - CD-- 297/3/0/0/0 0/0 "GET
/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file HTTP/1.1"
The glance-api server logs the following 'Connection reset by peer'
error, same as openstack cmdline client.
node001: 2021-12-10 11:52:15.247 58 INFO eventlet.wsgi.server [req-4af3af95-6363-4857-ad82-7a30c784bc40 a8814484a71c4c5193282bcfd5e6ff3f fea7b584dcc343c19c65498e099ffd80 - default default] Traceback (most recent call last):
node001: File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 591, in handle_one_response
node001: write(b''.join(towrite))
node001: File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 537, in write
node001: wfile.writelines(towrite)
node001: File "/usr/lib64/python3.6/socket.py", line 604, in write
node001: return self._sock.send(b)
node001: File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 397, in send
node001: return self._send_loop(self.fd.send, data, flags)
node001: File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
node001: return send_method(data, *args)
node001: ConnectionResetError: [Errno 104] Connection reset by peer
Can this issue be resolved by modifying the default haproxy configuration?
What you expected to happen:
The 'openstack image create' and 'openstack image save' commands
should execute succesfully.
How to reproduce it (minimal and precise):
Try to upload a 40GB+ image with 'openstack image create' command, in
a Kolla Victoria setup, in High availability mode, with Haproxy and
virtual IPs.
**Environment**:
* OS : CentOS Linux 8
* Kernel : 4.18.0-305.3.1.el8.x86_64
* Docker version : 20.10.8(Docker Engine - Community)
* Kolla-ansible version : 11.1.0
* Openstack version : Victoria
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1954430/+subscriptions
Follow ups
