yahoo-eng-team team mailing list archive

[Alvaro Lopez <1361184@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

There's a race condition in imagebackend.Image.cache that makes nova
download an image N times when N requests requiring the same image are
scheduled in the same host during the image feching.

The imagebackend.Image.cache method only synchronizes on the image
fetching function, but the whole function should be synchronized (or the
create_image function). When several requests using the same image are
scheduled at the same time there's no synchronization when nova checks
if an image already exists or not, therefore several requests may check
that the image does not exist, and start a download for all of them (the
actual download will be syncronized, but the image will be download
several times, one for each request).

This can be seen requesting several instances into the same host:

    nova boot --image <image> --flavor <foo> --num-instances=4
--availability-zone <az>:<host>

In the host we can see:

    -rw-r--r-- 1 nova nova 5.0G Aug 25 14:21 243eccfbc52469947665a506145d798670e3fc88
    -rw-r--r-- 1 nova nova 1.2G Aug 25 14:22 243eccfbc52469947665a506145d798670e3fc88.part

** Affects: nova
     Importance: Undecided
     Assignee: Alvaro Lopez (aloga)
         Status: In Progress

** Changed in: nova
       Status: New => In Progress

** Changed in: nova
     Assignee: (unassigned) => Alvaro Lopez (aloga)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1361184

Title:
  Race condition in imagebackend.Image.cache downloads image several
  times

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  There's a race condition in imagebackend.Image.cache that makes nova
  download an image N times when N requests requiring the same image are
  scheduled in the same host during the image feching.

  The imagebackend.Image.cache method only synchronizes on the image
  fetching function, but the whole function should be synchronized (or
  the create_image function). When several requests using the same image
  are scheduled at the same time there's no synchronization when nova
  checks if an image already exists or not, therefore several requests
  may check that the image does not exist, and start a download for all
  of them (the actual download will be syncronized, but the image will
  be download several times, one for each request).

  This can be seen requesting several instances into the same host:

      nova boot --image <image> --flavor <foo> --num-instances=4
  --availability-zone <az>:<host>

  In the host we can see:

      -rw-r--r-- 1 nova nova 5.0G Aug 25 14:21 243eccfbc52469947665a506145d798670e3fc88
      -rw-r--r-- 1 nova nova 1.2G Aug 25 14:22 243eccfbc52469947665a506145d798670e3fc88.part

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1361184/+subscriptions