yahoo-eng-team team mailing list archive

[Vipul Nayyar <1954430@xxxxxxxxxxxxxxxxxx>]

The above issue is caused since kolla-ansible and ceph-ansible both use
the same router_id('51') for keepalived configuration. A different
router id for kolla-ansible can be set with
'keepalived_virtual_router_id' option in globals.yml.

This issue can be marked as solved now.

** No longer affects: glance

** Changed in: kolla
       Status: New => Incomplete

** Changed in: kolla
       Status: Incomplete => Opinion

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1954430

Title:
  Openstack image command-line operations fail for large images in Kolla
  Victoria

Status in kolla:
  Opinion

Bug description:
  **Bug Report**

  What happened:

  Openstack image upload and save commands fail with the following
  'Connection reset by peer' error, for only large images.

  # openstack image save windows-server2016 --file /tmp/t --debug

  GET call to image for http://172.16.201.254:9292/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file used request id req-d15c2664-b776-4486-8f12-fb0eb2bdfef8
  Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 438, in _error_catcher
      yield
    File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 519, in read
      data = self._fp.read(amt) if not fp_closed else b""
    File "/usr/lib64/python3.6/http/client.py", line 463, in read
      n = self.readinto(b)
    File "/usr/lib64/python3.6/http/client.py", line 507, in readinto
      n = self.fp.readinto(b)
    File "/usr/lib64/python3.6/socket.py", line 586, in readinto
      return self._sock.recv_into(b)
  ConnectionResetError: [Errno 104] Connection reset by peer

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 758, in generate
      for chunk in self.raw.stream(chunk_size, decode_content=True):
    File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 576, in stream
      data = self.read(amt=amt, decode_content=decode_content)
    File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 541, in read
      raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
    File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
      self.gen.throw(type, value, traceback)
    File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 455, in _error_catcher
      raise ProtocolError("Connection broken: %r" % e, e)
  urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
      chunk_size=chunk_size):
    File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 761, in generate
      raise ChunkedEncodingError(e)
  requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
      result = cmd.run(parsed_args)
    File "/usr/local/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
      return super(Command, self).run(parsed_args)
    File "/usr/local/lib/python3.6/site-packages/cliff/command.py", line 186, in run
      return_code = self.take_action(parsed_args) or 0
    File "/usr/local/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 847, in take_action
      image_client.download_image(image.id, stream=True, output=output_file)
    File "/usr/local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
      self, stream=stream, output=output, chunk_size=chunk_size)
    File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
      "Unable to download image: %s" % e)
  openstack.exceptions.SDKException: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  clean_up SaveImage: Unable to download image: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
  END return value: 1


  The haproxy logs show a 'CD--' status, which suggest that the client
  aborted the connection.

  Dec 10 11:46:58 node003 haproxy[43]: 172.16.201.1:51626
  [10/Dec/2021:11:46:47.358] glance_api_front glance_api_back/node001
  0/0/0/219/11319 200 1132157897 - - CD-- 297/3/0/0/0 0/0 "GET
  /v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file HTTP/1.1"


  The glance-api server logs  the following 'Connection reset by peer'
  error, same as openstack cmdline client.

  node001: 2021-12-10 11:52:15.247 58 INFO eventlet.wsgi.server [req-4af3af95-6363-4857-ad82-7a30c784bc40 a8814484a71c4c5193282bcfd5e6ff3f fea7b584dcc343c19c65498e099ffd80 - default default] Traceback (most recent call last):
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 591, in handle_one_response
  node001:     write(b''.join(towrite))
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 537, in write
  node001:     wfile.writelines(towrite)
  node001:   File "/usr/lib64/python3.6/socket.py", line 604, in write
  node001:     return self._sock.send(b)
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 397, in send
  node001:     return self._send_loop(self.fd.send, data, flags)
  node001:   File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
  node001:     return send_method(data, *args)
  node001: ConnectionResetError: [Errno 104] Connection reset by peer


  
  Can this issue be resolved by modifying the default haproxy configuration?


  What you expected to happen:

  The 'openstack image create' and 'openstack image save' commands
  should execute succesfully.

  How to reproduce it (minimal and precise):

  Try to upload a 40GB+ image with 'openstack image create' command, in
  a Kolla Victoria setup, in High availability mode, with Haproxy and
  virtual IPs.

  **Environment**:

  * OS : CentOS Linux 8
  * Kernel : 4.18.0-305.3.1.el8.x86_64
  * Docker version : 20.10.8(Docker Engine - Community)
  * Kolla-ansible version : 11.1.0
  * Openstack version : Victoria

To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla/+bug/1954430/+subscriptions