yahoo-eng-team team mailing list archive

[Alexander Tivelkov <ativelkov@xxxxxxxxxxxx>]

Public bug reported:

UpdateImage call in v2 ( PATCH /v2/images/%imageid%) fetches the whole
image object from database
(https://github.com/openstack/glance/blob/master/glance/api/v2/images.py#L119),
modifies some of its properties with appropriate matching methods
(_do_replace, _do_add etc) and then saves the object in ImageRepo

The imageRepo takes the object, renders it to dictionary
(https://github.com/openstack/glance/blob/master/glance/db/__init__.py#L169)
and then passes this dictionary to db_api.image_update(), which
eventually executes UPDATE query and replaces all the fields in database
with these values.

This is not transactional, and other operations may interfere between
image fetching and update query.

For example, imagine we have an image with name "test" and min_ram "10".
If we concurrently execute two image-update requests:
  'image-update --name test2'
and
  'image-update --min-ram 100'

A race condition occurs. The result image is undetermined: it may be either {'name': 'test', 'min_ram': 100} or {'name': 'test2', 'min_ram': 10} or  {'name': 'test2', 'min_ram': 100'} - depending on which request fetches the object first.
It may be had to reproduce this in normal circumstances, but once the database load becomes higher, it may show up.
To reproduce it for sure one may add a event let.sleep() command after line 119 of glance/api/v2/images.py, to increase the timespan between image fetching and image update.

** Affects: glance
     Importance: Undecided
     Assignee: Alexander Tivelkov (ativelkov)
         Status: New

** Description changed:

  UpdateImage call in v2 ( PATCH /v2/images/%imageid%) fetches the whole
  image object from database
  (https://github.com/openstack/glance/blob/master/glance/api/v2/images.py#L119),
  modifies some of its properties with appropriate matching methods
- (_do_raplace, _do_add etc) and then saves the object in ImageRepo
+ (_do_replace, _do_add etc) and then saves the object in ImageRepo
  
  The imageRepo takes the object, renders it to dictionary
  (https://github.com/openstack/glance/blob/master/glance/db/__init__.py#L169)
  and then passes this dictionary to db_api.image_update(), which
  eventually executes UPDATE query and replaces all the fields in database
  with these values.
  
  This is not transactional, and other operations may interfere between
  image fetching and update query.
  
- For example, imagine we have an image with name "test" and min_ram "10". 
+ For example, imagine we have an image with name "test" and min_ram "10".
  If we concurrently execute two image-update requests:
-   'image-update --name test2'
+   'image-update --name test2'
  and
-   'image-update --min-ram 100'
+   'image-update --min-ram 100'
  
  A race condition occurs. The result image is undetermined: it may be either {'name': 'test', 'min_ram': 100} or {'name': 'test2', 'min_ram': 10} or  {'name': 'test2', 'min_ram': 100'} - depending on which request fetches the object first.
  It may be had to reproduce this in normal circumstances, but once the database load becomes higher, it may show up.
  To reproduce it for sure one may add a event let.sleep() command after line 119 of glance/api/v2/images.py, to increase the timespan between image fetching and image update.

** Changed in: glance
     Assignee: (unassigned) => Alexander Tivelkov (ativelkov)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1371728

Title:
  Concurrency Update issue in v2

Status in OpenStack Image Registry and Delivery Service (Glance):
  New

Bug description:
  UpdateImage call in v2 ( PATCH /v2/images/%imageid%) fetches the whole
  image object from database
  (https://github.com/openstack/glance/blob/master/glance/api/v2/images.py#L119),
  modifies some of its properties with appropriate matching methods
  (_do_replace, _do_add etc) and then saves the object in ImageRepo

  The imageRepo takes the object, renders it to dictionary
  (https://github.com/openstack/glance/blob/master/glance/db/__init__.py#L169)
  and then passes this dictionary to db_api.image_update(), which
  eventually executes UPDATE query and replaces all the fields in
  database with these values.

  This is not transactional, and other operations may interfere between
  image fetching and update query.

  For example, imagine we have an image with name "test" and min_ram "10".
  If we concurrently execute two image-update requests:
    'image-update --name test2'
  and
    'image-update --min-ram 100'

  A race condition occurs. The result image is undetermined: it may be either {'name': 'test', 'min_ram': 100} or {'name': 'test2', 'min_ram': 10} or  {'name': 'test2', 'min_ram': 100'} - depending on which request fetches the object first.
  It may be had to reproduce this in normal circumstances, but once the database load becomes higher, it may show up.
  To reproduce it for sure one may add a event let.sleep() command after line 119 of glance/api/v2/images.py, to increase the timespan between image fetching and image update.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1371728/+subscriptions