yahoo-eng-team team mailing list archive

[Alexander Tivelkov <ativelkov@xxxxxxxxxxxx>]

Public bug reported:

Glance may loose some image data while transferring it to the backing
store thus corrupting the image when ALL the following conditions are
met:

- Image is being created by copying data from remote source (--copy-from CLI parameter or appropriate API call)
- Backing store is Swift
- Image size is larger then configured "swift_store_large_object_size"

In such scenarios the last chunk stored in Swift will have the size
significantly less then expected. An attempt to download the image will
result in a checksum verification error, however the checksum stored in
Glance (with image metadata) is correct, and so is the size.

This is easily reproducible even on devstack (if the devstack is
configured to run Swift as Glance backend). Just decrease
'swift_store_large_object_size' to some reasonably low value (i.e. 200
Mb) and try to copy-from any image which is larger then that value.
After the upload is successful, check the object size in swift (by
either summing the sizes of all the chunks or by looking to the size of
virtual large object) - they will be lower then expected:


glance image-create --name tst --disk-format qcow2 --container-format bare --copy-from http://192.168.56.1:8000/F18-x86_64-cfntools.qcow2

...

glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                            | Disk Format | Container Format | Size      | Status |
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
| fc34ec49-4bd3-40dd-918f-44d3254f2ac9 | tst                             | qcow2       | bare             | 536412160 | active |
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+

...

swift stat glance fc34ec49-4bd3-40dd-918f-44d3254f2ac9 --os-tenant-name service --os-username admin
       Account: AUTH_cce6e9c12fa34c63b64ef29e84861554
     Container: glance
        Object: fc34ec49-4bd3-40dd-918f-44d3254f2ac9
  Content Type: application/octet-stream
Content Length: 509804544                         <---- see, the size is different!
 Last Modified: Mon, 19 Jan 2015 15:52:18 GMT
          ETag: "6d0612f82db9a531b34d25823a45073d"
      Manifest: glance/fc34ec49-4bd3-40dd-918f-44d3254f2ac9-
 Accept-Ranges: bytes
   X-Timestamp: 1421682737.01148
    X-Trans-Id: tx01a19f7476a541808c9a1-0054bd28e1

....

glance image-download tst --file out.qcow2
[Errno 32] Corrupt image download. Checksum was 0eeddae1007f01b0029136d28518f538 expected 3ecddfe0787a392960d230c87a421c6a

** Affects: glance
     Importance: Undecided
     Assignee: Alexander Tivelkov (ativelkov)
         Status: New

** Changed in: glance
     Assignee: (unassigned) => Alexander Tivelkov (ativelkov)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1412802

Title:
  copy-from broken for large files and swift

Status in OpenStack Image Registry and Delivery Service (Glance):
  New

Bug description:
  Glance may loose some image data while transferring it to the backing
  store thus corrupting the image when ALL the following conditions are
  met:

  - Image is being created by copying data from remote source (--copy-from CLI parameter or appropriate API call)
  - Backing store is Swift
  - Image size is larger then configured "swift_store_large_object_size"

  In such scenarios the last chunk stored in Swift will have the size
  significantly less then expected. An attempt to download the image
  will result in a checksum verification error, however the checksum
  stored in Glance (with image metadata) is correct, and so is the size.

  This is easily reproducible even on devstack (if the devstack is
  configured to run Swift as Glance backend). Just decrease
  'swift_store_large_object_size' to some reasonably low value (i.e. 200
  Mb) and try to copy-from any image which is larger then that value.
  After the upload is successful, check the object size in swift (by
  either summing the sizes of all the chunks or by looking to the size
  of virtual large object) - they will be lower then expected:

  
  glance image-create --name tst --disk-format qcow2 --container-format bare --copy-from http://192.168.56.1:8000/F18-x86_64-cfntools.qcow2

  ...

  glance image-list
  +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
  | ID                                   | Name                            | Disk Format | Container Format | Size      | Status |
  +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
  | fc34ec49-4bd3-40dd-918f-44d3254f2ac9 | tst                             | qcow2       | bare             | 536412160 | active |
  +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+

  ...

  swift stat glance fc34ec49-4bd3-40dd-918f-44d3254f2ac9 --os-tenant-name service --os-username admin
         Account: AUTH_cce6e9c12fa34c63b64ef29e84861554
       Container: glance
          Object: fc34ec49-4bd3-40dd-918f-44d3254f2ac9
    Content Type: application/octet-stream
  Content Length: 509804544                         <---- see, the size is different!
   Last Modified: Mon, 19 Jan 2015 15:52:18 GMT
            ETag: "6d0612f82db9a531b34d25823a45073d"
        Manifest: glance/fc34ec49-4bd3-40dd-918f-44d3254f2ac9-
   Accept-Ranges: bytes
     X-Timestamp: 1421682737.01148
      X-Trans-Id: tx01a19f7476a541808c9a1-0054bd28e1

  ....

  glance image-download tst --file out.qcow2
  [Errno 32] Corrupt image download. Checksum was 0eeddae1007f01b0029136d28518f538 expected 3ecddfe0787a392960d230c87a421c6a

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1412802/+subscriptions