← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #96620

[Bug 2127521] [NEW] read function in class _UWSGIChunkFile can return more data than requested

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

The typical behavior of the read function which has length in the parameter is to return data <= that length.
It's similar to read system call that has count parameter and it's not expected behavior that the function returns more.
There are some places that assume that, e.g. class ChunkReader(object) [1].
it doesn't expect that the len(self.do_read(i)) can be bigger than i.

But read function in class _UWSGIChunkFile can return more data than requested.
Due to announced deprecation of eventlets in glance we switched to uwsgi. We use swift backend for glance in our setup.
When we tried making snapshot of the VM, created from 2.01 GB QCOW2 image with Create New Volume = No, we started getting really high consumption of memory which caused OOM.
During the investigation we found that _UWSGIChunkFile.read returns sometimes more data than it was requested (by adding debug outputs into ChunkReader [2]. Because of this behavior ChunkReader gets negative value of i (see the last 2 lines) and in its turn the last call causes _UWSGIChunkFile to read the whole remanings of the file into memory (~5GB). IF we increase OOM limit, glance tries to send this 5+Gb to swift and gets error, because swift typically has limit and under normal condition it sends data using 200Mb (swift_store_large_object_chunk_size can change that limit).

Before switching to uwsgi we didn't see that issue and the data was
transferred by 200Mb.


[1]
https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_drivers/swift/store.py#L1672

[2]
    def read(self, i):

        left = self.total - self.bytes_read

        if i > left:

            i = left

...
        if i == 0:

            result = b''

        else:

            result = self.do_read(i)


        LOG.info(("read i: %d, res: %d" % (i, len(result))))

        self.bytes_read += len(result)

        self.checksum.update(result)

        self.os_hash_value.update(result)

        if self.verifier:

            self.verifier.update(result)

we see:
2025-10-10 01:34:28.291 7 INFO glance_store._drivers.swift.store [None req-b7061fc2-d029-441f-aef8-2a7d5efe088b 1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - - default default] read i: 65536, res: 73720
...
2025-10-10 01:34:31.712 7 INFO glance_store._drivers.swift.store [None req-b7061fc2-d029-441f-aef8-2a7d5efe088b 1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - - default default] read i: 34726, res: 40960

2025-10-10 01:35:00.645 7 INFO glance_store._drivers.swift.store [None
req-b7061fc2-d029-441f-aef8-2a7d5efe088b
1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - -
default default] read i: -6234, res: 5053605798

** Affects: glance
     Importance: Undecided
         Status: New

** Also affects: glance
   Importance: Undecided
       Status: New

** No longer affects: python-openstackclient

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2127521

Title:
  read function in class _UWSGIChunkFile can return more data than
  requested

Status in Glance:
  New

Bug description:
  The typical behavior of the read function which has length in the parameter is to return data <= that length.
  It's similar to read system call that has count parameter and it's not expected behavior that the function returns more.
  There are some places that assume that, e.g. class ChunkReader(object) [1].
  it doesn't expect that the len(self.do_read(i)) can be bigger than i.

  But read function in class _UWSGIChunkFile can return more data than requested.
  Due to announced deprecation of eventlets in glance we switched to uwsgi. We use swift backend for glance in our setup.
  When we tried making snapshot of the VM, created from 2.01 GB QCOW2 image with Create New Volume = No, we started getting really high consumption of memory which caused OOM.
  During the investigation we found that _UWSGIChunkFile.read returns sometimes more data than it was requested (by adding debug outputs into ChunkReader [2]. Because of this behavior ChunkReader gets negative value of i (see the last 2 lines) and in its turn the last call causes _UWSGIChunkFile to read the whole remanings of the file into memory (~5GB). IF we increase OOM limit, glance tries to send this 5+Gb to swift and gets error, because swift typically has limit and under normal condition it sends data using 200Mb (swift_store_large_object_chunk_size can change that limit).

  Before switching to uwsgi we didn't see that issue and the data was
  transferred by 200Mb.

  
  [1]
  https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_drivers/swift/store.py#L1672

  [2]
      def read(self, i):

          left = self.total - self.bytes_read

          if i > left:

              i = left

  ...
          if i == 0:

              result = b''

          else:

              result = self.do_read(i)


          LOG.info(("read i: %d, res: %d" % (i, len(result))))

          self.bytes_read += len(result)

          self.checksum.update(result)

          self.os_hash_value.update(result)

          if self.verifier:

              self.verifier.update(result)

  we see:
  2025-10-10 01:34:28.291 7 INFO glance_store._drivers.swift.store [None req-b7061fc2-d029-441f-aef8-2a7d5efe088b 1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - - default default] read i: 65536, res: 73720
  ...
  2025-10-10 01:34:31.712 7 INFO glance_store._drivers.swift.store [None req-b7061fc2-d029-441f-aef8-2a7d5efe088b 1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - - default default] read i: 34726, res: 40960

  2025-10-10 01:35:00.645 7 INFO glance_store._drivers.swift.store [None
  req-b7061fc2-d029-441f-aef8-2a7d5efe088b
  1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - -
  default default] read i: -6234, res: 5053605798

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2127521/+subscriptions
  Thread Previous • Date Previous • • Thread Next